Nov 03 22:12:17.183508 ip-10-0-139-73 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 22:12:17.541242 ip-10-0-139-73 kubenswrapper[2570]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 22:12:17.541242 ip-10-0-139-73 kubenswrapper[2570]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 22:12:17.541242 ip-10-0-139-73 kubenswrapper[2570]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 22:12:17.541242 ip-10-0-139-73 kubenswrapper[2570]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 22:12:17.544550 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544468    2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 22:12:17.544550 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544549    2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 22:12:17.544550 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544553    2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 22:12:17.544720 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544556    2570 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 22:12:17.544720 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544559    2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 22:12:17.544720 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544561    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 22:12:17.544720 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544564    2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 22:12:17.544720 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544568    2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 22:12:17.544720 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544571    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 22:12:17.544720 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544574    2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 22:12:17.544720 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544576    2570 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 22:12:17.544720 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544579    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 22:12:17.544720 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544582    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 22:12:17.544720 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544584    2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 22:12:17.544720 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544587    2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 22:12:17.544720 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544589    2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 22:12:17.544720 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544592    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 22:12:17.544720 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544595    2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 22:12:17.544720 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544599    2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 22:12:17.544720 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544604    2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 22:12:17.544720 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544607    2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 22:12:17.544720 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544609    2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 22:12:17.545157 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544613    2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 22:12:17.545157 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544616    2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 22:12:17.545157 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544619    2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 22:12:17.545157 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544621    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 22:12:17.545157 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544624    2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 22:12:17.545157 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544627    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 22:12:17.545157 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544631    2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 22:12:17.545157 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544634    2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 22:12:17.545157 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544637    2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 22:12:17.545157 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544639    2570 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 22:12:17.545157 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544642    2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 22:12:17.545157 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544645    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 22:12:17.545157 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544648    2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 22:12:17.545157 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544650    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 22:12:17.545157 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544653    2570 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 22:12:17.545157 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544655    2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 22:12:17.545157 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544658    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 22:12:17.545157 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544661    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 22:12:17.545157 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544663    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 22:12:17.545157 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544666    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 22:12:17.545647 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544668    2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 22:12:17.545647 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544672    2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 22:12:17.545647 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544674    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 22:12:17.545647 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544676    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 22:12:17.545647 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544679    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 22:12:17.545647 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544682    2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 22:12:17.545647 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544684    2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 22:12:17.545647 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544687    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 22:12:17.545647 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544689    2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 22:12:17.545647 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544692    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 22:12:17.545647 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544694    2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 22:12:17.545647 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544697    2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 22:12:17.545647 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544700    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 22:12:17.545647 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544702    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 22:12:17.545647 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544705    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 22:12:17.545647 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544708    2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 22:12:17.545647 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544710    2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 22:12:17.545647 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544713    2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 22:12:17.545647 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544716    2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 22:12:17.546087 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544719    2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 22:12:17.546087 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544722    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 22:12:17.546087 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544726    2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 22:12:17.546087 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544731    2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 22:12:17.546087 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544733    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 22:12:17.546087 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544736    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 22:12:17.546087 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544739    2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 22:12:17.546087 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544742    2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 22:12:17.546087 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544751    2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 22:12:17.546087 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544754    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 22:12:17.546087 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544757    2570 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 22:12:17.546087 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544760    2570 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 22:12:17.546087 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544763    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 22:12:17.546087 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544765    2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 22:12:17.546087 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544768    2570 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 22:12:17.546087 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544771    2570 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 22:12:17.546087 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544773    2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 22:12:17.546087 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544776    2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 22:12:17.546087 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544778    2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 22:12:17.546087 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544781    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 22:12:17.546870 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544783    2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 22:12:17.546870 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544786    2570 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 22:12:17.546870 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544788    2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 22:12:17.546870 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544791    2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 22:12:17.546870 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544793    2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 22:12:17.546870 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544796    2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 22:12:17.546870 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544798    2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 22:12:17.546870 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544801    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 22:12:17.546870 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544803    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 22:12:17.546870 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544806    2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 22:12:17.546870 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544809    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 22:12:17.546870 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544812    2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 22:12:17.546870 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544815    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 22:12:17.546870 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544817    2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 22:12:17.546870 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544820    2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 22:12:17.546870 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544823    2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 22:12:17.546870 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544826    2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 22:12:17.546870 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544830    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 22:12:17.546870 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544832    2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 22:12:17.547411 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544835    2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 22:12:17.547411 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544838    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 22:12:17.547411 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544840    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 22:12:17.547411 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544843    2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 22:12:17.547411 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544846    2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 22:12:17.547411 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.544849    2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 22:12:17.547411 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.546580    2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 22:12:17.547411 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.546596    2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 22:12:17.547411 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.546602    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 22:12:17.547411 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.546612    2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 22:12:17.547411 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.546617    2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 22:12:17.547411 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.546621    2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 22:12:17.547411 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.546626    2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 22:12:17.547411 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.546630    2570 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 22:12:17.547411 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.546635    2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 22:12:17.547411 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.546639    2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 22:12:17.547411 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.546643    2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 22:12:17.547411 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.546648    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 22:12:17.547411 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.546652    2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 22:12:17.547852 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.546656    2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 22:12:17.547852 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.546660    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 22:12:17.547852 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.546664    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 22:12:17.547852 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.546673    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 22:12:17.547852 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.546678    2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 22:12:17.547852 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.546683    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 22:12:17.547852 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.546687    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 22:12:17.547852 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.546691    2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 22:12:17.547852 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.546696    2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 22:12:17.547852 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.546700    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 22:12:17.547852 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.546704    2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 22:12:17.547852 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.546708    2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 22:12:17.547852 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.546713    2570 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 22:12:17.547852 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.546717    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 22:12:17.547852 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.546721    2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 22:12:17.547852 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.546726    2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 22:12:17.547852 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.546734    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 22:12:17.547852 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.546739    2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 22:12:17.547852 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.546743    2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 22:12:17.548336 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.546747    2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 22:12:17.548336 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.546751    2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 22:12:17.548336 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.546756    2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 22:12:17.548336 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.546760    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 22:12:17.548336 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.546764    2570 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 22:12:17.548336 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.546768    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 22:12:17.548336 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.546772    2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 22:12:17.548336 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.546776    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 22:12:17.548336 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.546780    2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 22:12:17.548336 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.546789    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 22:12:17.548336 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.546794    2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 22:12:17.548336 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.546798    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 22:12:17.548336 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.546802    2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 22:12:17.548336 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.546806    2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 22:12:17.548336 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.546810    2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 22:12:17.548336 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.546814    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 22:12:17.548336 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.546818    2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 22:12:17.548336 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.546824    2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 22:12:17.548336 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.546828    2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 22:12:17.548336 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.546833    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 22:12:17.548801 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.546837    2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 22:12:17.548801 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.546846    2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 22:12:17.548801 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.546858    2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 22:12:17.548801 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.546864    2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 22:12:17.548801 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.546868    2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 22:12:17.548801 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.546873    2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 22:12:17.548801 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.546877    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 22:12:17.548801 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.546882    2570 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 22:12:17.548801 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.546886    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 22:12:17.548801 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.546891    2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 22:12:17.548801 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.546895    2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 22:12:17.548801 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.546899    2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 22:12:17.548801 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.546904    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 22:12:17.548801 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.546908    2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 22:12:17.548801 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.546918    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 22:12:17.548801 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.546923    2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 22:12:17.548801 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.546929    2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 22:12:17.548801 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.546934    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 22:12:17.549237 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.546939    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 22:12:17.549237 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.546943    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 22:12:17.549237 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.546947    2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 22:12:17.549237 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.546951    2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 22:12:17.549237 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.546955    2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 22:12:17.549237 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.546963    2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 22:12:17.549237 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.546967    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 22:12:17.549237 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.546976    2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 22:12:17.549237 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.546981    2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 22:12:17.549237 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.546985    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 22:12:17.549237 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.546989    2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 22:12:17.549237 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.546993    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 22:12:17.549237 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.546997    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 22:12:17.549237 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.547002    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 22:12:17.549237 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.547006    2570 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 22:12:17.549237 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.547010    2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 22:12:17.549237 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.547015    2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 22:12:17.549237 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.547020    2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 22:12:17.549237 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.547025    2570 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 22:12:17.549691 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.547029    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 22:12:17.549691 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.547038    2570 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 22:12:17.549691 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.547043    2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 22:12:17.549691 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.547047    2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 22:12:17.549691 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.547051    2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 22:12:17.549691 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.547055    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 22:12:17.549691 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.547059    2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 22:12:17.549691 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.547064    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 22:12:17.549691 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.547068    2570 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 22:12:17.549691 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.547072    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 22:12:17.549691 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.547154    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 22:12:17.549691 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.547184    2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 22:12:17.549691 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.547313    2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 22:12:17.549691 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.547319    2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 22:12:17.549691 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.547324    2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 22:12:17.549691 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.547328    2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 22:12:17.549691 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.547332    2570 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 22:12:17.549691 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.547906    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 22:12:17.549691 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.547914    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 22:12:17.549691 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.547918    2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 22:12:17.550159 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.547921    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 22:12:17.550159 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.547924    2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 22:12:17.550159 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.547926    2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 22:12:17.550159 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.547930    2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 22:12:17.550159 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.547935    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 22:12:17.550159 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.547939    2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 22:12:17.550159 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.547942    2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 22:12:17.550159 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.547945    2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 22:12:17.550159 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.547947    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 22:12:17.550159 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.547949    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 22:12:17.550159 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.547952    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 22:12:17.550159 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.547954    2570 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 22:12:17.550159 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.547958    2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 22:12:17.550159 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.547960    2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 22:12:17.550159 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.547963    2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 22:12:17.550159 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.547966    2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 22:12:17.550159 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.547969    2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 22:12:17.550159 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.547971    2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 22:12:17.550159 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.547974    2570 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 22:12:17.550159 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.547977    2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 22:12:17.550643 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.547980    2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 22:12:17.550643 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.547982    2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 22:12:17.550643 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.547985    2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 22:12:17.550643 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.547988    2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 22:12:17.550643 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.547991    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 22:12:17.550643 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.547994    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 22:12:17.550643 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.547997    2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 22:12:17.550643 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.547999    2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 22:12:17.550643 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.548002    2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 22:12:17.550643 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.548005    2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 22:12:17.550643 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.548008    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 22:12:17.550643 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.548010    2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 22:12:17.550643 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.548013    2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 22:12:17.550643 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.548015    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 22:12:17.550643 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.548018    2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 22:12:17.550643 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.548020    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 22:12:17.550643 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.548023    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 22:12:17.550643 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.548025    2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 22:12:17.550643 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.548028    2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 22:12:17.550643 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.548030    2570 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 22:12:17.551132 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.548033    2570 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 22:12:17.551132 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.548045    2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 22:12:17.551132 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.548048    2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 22:12:17.551132 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.548050    2570 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 22:12:17.551132 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.548053    2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 22:12:17.551132 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.548056    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 22:12:17.551132 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.548059    2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 22:12:17.551132 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.548062    2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 22:12:17.551132 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.548064    2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 22:12:17.551132 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.548067    2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 22:12:17.551132 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.548069    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 22:12:17.551132 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.548072    2570 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 22:12:17.551132 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.548074    2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 22:12:17.551132 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.548079    2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 22:12:17.551132 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.548083    2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 22:12:17.551132 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.548088    2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 22:12:17.551132 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.548092    2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 22:12:17.551132 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.548095    2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 22:12:17.551132 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.548097    2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 22:12:17.551589 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.548100    2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 22:12:17.551589 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.548103    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 22:12:17.551589 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.548106    2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 22:12:17.551589 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.548109    2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 22:12:17.551589 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.548112    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 22:12:17.551589 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.548114    2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 22:12:17.551589 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.548116    2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 22:12:17.551589 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.548119    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 22:12:17.551589 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.548122    2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 22:12:17.551589 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.548124    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 22:12:17.551589 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.548127    2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 22:12:17.551589 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.548129    2570 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 22:12:17.551589 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.548132    2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 22:12:17.551589 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.548134    2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 22:12:17.551589 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.548137    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 22:12:17.551589 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.548140    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 22:12:17.551589 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.548142    2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 22:12:17.551589 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.548145    2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 22:12:17.551589 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.548148    2570 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 22:12:17.552044 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.548150    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 22:12:17.552044 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.548153    2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 22:12:17.552044 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.548155    2570 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 22:12:17.552044 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.548157    2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 22:12:17.552044 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.548160    2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 22:12:17.552044 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.548162    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 22:12:17.552044 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.548166    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 22:12:17.552044 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.548168    2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 22:12:17.552044 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.548173    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 22:12:17.552044 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.548176    2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 22:12:17.552044 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.548178    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 22:12:17.552044 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.548181    2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 22:12:17.552044 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.548184    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 22:12:17.552044 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.548186    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 22:12:17.552044 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.548188    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 22:12:17.552044 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.548191    2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 22:12:17.552044 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.548194    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 22:12:17.552044 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.548196    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 22:12:17.552044 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.548198    2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 22:12:17.552549 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.548201    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 22:12:17.552549 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.548204    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 22:12:17.552549 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.548206    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 22:12:17.552549 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.548209    2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 22:12:17.552549 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.548225    2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 22:12:17.552549 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.548230    2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 22:12:17.552549 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.549899    2570 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 22:12:17.552549 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.549911    2570 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 22:12:17.552549 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.549919    2570 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 22:12:17.552549 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.549924    2570 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 22:12:17.552549 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.549930    2570 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 22:12:17.552549 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.549934    2570 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 22:12:17.552549 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.549938    2570 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 22:12:17.552549 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.549943    2570 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 22:12:17.552549 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.549946    2570 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 22:12:17.552549 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.549950    2570 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 22:12:17.552549 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.549954    2570 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 22:12:17.552549 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.549957    2570 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 22:12:17.552549 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.549960    2570 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 22:12:17.552549 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.549964    2570 flags.go:64] FLAG: --cgroup-root=""
Nov 03 22:12:17.552549 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.549967    2570 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 22:12:17.552549 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.549970    2570 flags.go:64] FLAG: --client-ca-file=""
Nov 03 22:12:17.553049 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.549974    2570 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 22:12:17.553049 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.549977    2570 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 22:12:17.553049 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.549981    2570 flags.go:64] FLAG: --cluster-domain=""
Nov 03 22:12:17.553049 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.549984    2570 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 22:12:17.553049 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.549988    2570 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 22:12:17.553049 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.549991    2570 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 22:12:17.553049 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.549994    2570 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 22:12:17.553049 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.549998    2570 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 22:12:17.553049 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550001    2570 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 22:12:17.553049 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550004    2570 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 22:12:17.553049 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550008    2570 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 22:12:17.553049 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550011    2570 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 22:12:17.553049 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550014    2570 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 22:12:17.553049 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550017    2570 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 22:12:17.553049 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550021    2570 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 22:12:17.553049 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550024    2570 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 22:12:17.553049 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550028    2570 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 22:12:17.553049 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550032    2570 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 22:12:17.553049 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550035    2570 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 22:12:17.553049 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550038    2570 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 22:12:17.553049 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550041    2570 flags.go:64] FLAG: --enable-server="true"
Nov 03 22:12:17.553049 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550044    2570 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 22:12:17.553049 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550049    2570 flags.go:64] FLAG: --event-burst="100"
Nov 03 22:12:17.553049 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550052    2570 flags.go:64] FLAG: --event-qps="50"
Nov 03 22:12:17.553049 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550055    2570 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 22:12:17.553665 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550058    2570 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 22:12:17.553665 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550061    2570 flags.go:64] FLAG: --eviction-hard=""
Nov 03 22:12:17.553665 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550066    2570 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 22:12:17.553665 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550069    2570 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 22:12:17.553665 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550072    2570 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 22:12:17.553665 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550075    2570 flags.go:64] FLAG: --eviction-soft=""
Nov 03 22:12:17.553665 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550078    2570 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 22:12:17.553665 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550082    2570 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 22:12:17.553665 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550085    2570 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 22:12:17.553665 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550088    2570 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 22:12:17.553665 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550092    2570 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 22:12:17.553665 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550095    2570 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 22:12:17.553665 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550098    2570 flags.go:64] FLAG: --feature-gates=""
Nov 03 22:12:17.553665 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550102    2570 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 22:12:17.553665 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550105    2570 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 22:12:17.553665 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550109    2570 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 22:12:17.553665 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550112    2570 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 22:12:17.553665 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550115    2570 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 22:12:17.553665 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550118    2570 flags.go:64] FLAG: --help="false"
Nov 03 22:12:17.553665 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550122    2570 flags.go:64] FLAG: --hostname-override="ip-10-0-139-73.ec2.internal"
Nov 03 22:12:17.553665 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550125    2570 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 22:12:17.553665 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550128    2570 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 22:12:17.553665 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550131    2570 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 22:12:17.554205 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550135    2570 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 22:12:17.554205 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550139    2570 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 22:12:17.554205 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550141    2570 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 22:12:17.554205 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550144    2570 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 22:12:17.554205 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550148    2570 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 22:12:17.554205 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550151    2570 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 22:12:17.554205 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550154    2570 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 22:12:17.554205 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550158    2570 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 22:12:17.554205 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550161    2570 flags.go:64] FLAG: --kube-reserved=""
Nov 03 22:12:17.554205 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550164    2570 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 22:12:17.554205 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550167    2570 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 22:12:17.554205 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550172    2570 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 22:12:17.554205 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550175    2570 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 22:12:17.554205 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550178    2570 flags.go:64] FLAG: --lock-file=""
Nov 03 22:12:17.554205 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550181    2570 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 22:12:17.554205 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550185    2570 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 22:12:17.554205 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550188    2570 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 22:12:17.554205 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550194    2570 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 22:12:17.554205 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550197    2570 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 22:12:17.554205 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550200    2570 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 22:12:17.554205 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550203    2570 flags.go:64] FLAG: --logging-format="text"
Nov 03 22:12:17.554205 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550206    2570 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 22:12:17.554205 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550209    2570 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 22:12:17.554205 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550224    2570 flags.go:64] FLAG: --manifest-url=""
Nov 03 22:12:17.554775 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550229    2570 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 22:12:17.554775 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550234    2570 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 22:12:17.554775 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550237    2570 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 22:12:17.554775 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550241    2570 flags.go:64] FLAG: --max-pods="110"
Nov 03 22:12:17.554775 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550244    2570 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 22:12:17.554775 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550247    2570 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 22:12:17.554775 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550251    2570 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 22:12:17.554775 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550254    2570 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 22:12:17.554775 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550257    2570 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 22:12:17.554775 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550260    2570 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 22:12:17.554775 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550263    2570 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 22:12:17.554775 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550271    2570 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 22:12:17.554775 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550274    2570 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 22:12:17.554775 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550278    2570 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 22:12:17.554775 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550281    2570 flags.go:64] FLAG: --pod-cidr=""
Nov 03 22:12:17.554775 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550284    2570 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 22:12:17.554775 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550287    2570 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 22:12:17.554775 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550290    2570 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 22:12:17.554775 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550293    2570 flags.go:64] FLAG: --port="10250"
Nov 03 22:12:17.554775 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550296    2570 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 22:12:17.554775 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550299    2570 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-09a331f3e94331be7"
Nov 03 22:12:17.554775 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550302    2570 flags.go:64] FLAG: --qos-reserved=""
Nov 03 22:12:17.554775 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550305    2570 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 22:12:17.554775 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550308    2570 flags.go:64] FLAG: --register-node="true"
Nov 03 22:12:17.554775 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550311    2570 flags.go:64] FLAG: --register-with-taints=""
Nov 03 22:12:17.554775 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550315    2570 flags.go:64] FLAG: --registry-burst="10"
Nov 03 22:12:17.555390 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550318    2570 flags.go:64] FLAG: --registry-qps="5"
Nov 03 22:12:17.555390 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550321    2570 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 22:12:17.555390 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550324    2570 flags.go:64] FLAG: --reserved-memory=""
Nov 03 22:12:17.555390 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550328    2570 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 22:12:17.555390 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550334    2570 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 22:12:17.555390 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550337    2570 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 22:12:17.555390 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550340    2570 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 22:12:17.555390 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550343    2570 flags.go:64] FLAG: --runonce="false"
Nov 03 22:12:17.555390 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550346    2570 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 22:12:17.555390 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550349    2570 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 22:12:17.555390 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550352    2570 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 22:12:17.555390 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550355    2570 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 22:12:17.555390 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550358    2570 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 22:12:17.555390 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550361    2570 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 22:12:17.555390 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550364    2570 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 22:12:17.555390 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550367    2570 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 22:12:17.555390 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550370    2570 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 22:12:17.555390 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550373    2570 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 22:12:17.555390 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550376    2570 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 22:12:17.555390 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550379    2570 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 22:12:17.555390 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550382    2570 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 22:12:17.555390 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550385    2570 flags.go:64] FLAG: --system-cgroups=""
Nov 03 22:12:17.555390 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550388    2570 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 22:12:17.555390 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550394    2570 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 22:12:17.555390 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550397    2570 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 22:12:17.555992 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550399    2570 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 22:12:17.555992 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550404    2570 flags.go:64] FLAG: --tls-min-version=""
Nov 03 22:12:17.555992 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550407    2570 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 22:12:17.555992 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550410    2570 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 22:12:17.555992 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550413    2570 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 22:12:17.555992 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550416    2570 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 22:12:17.555992 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550419    2570 flags.go:64] FLAG: --v="2"
Nov 03 22:12:17.555992 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550424    2570 flags.go:64] FLAG: --version="false"
Nov 03 22:12:17.555992 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550428    2570 flags.go:64] FLAG: --vmodule=""
Nov 03 22:12:17.555992 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550432    2570 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 22:12:17.555992 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550435    2570 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 22:12:17.555992 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550573    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 22:12:17.555992 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550576    2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 22:12:17.555992 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550579    2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 22:12:17.555992 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550582    2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 22:12:17.555992 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550585    2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 22:12:17.555992 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550587    2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 22:12:17.555992 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550590    2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 22:12:17.555992 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550595    2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 22:12:17.555992 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550598    2570 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 22:12:17.555992 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550601    2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 22:12:17.555992 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550604    2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 22:12:17.556533 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550607    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 22:12:17.556533 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550609    2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 22:12:17.556533 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550612    2570 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 22:12:17.556533 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550614    2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 22:12:17.556533 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550617    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 22:12:17.556533 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550619    2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 22:12:17.556533 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550623    2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 22:12:17.556533 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550625    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 22:12:17.556533 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550628    2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 22:12:17.556533 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550631    2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 22:12:17.556533 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550633    2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 22:12:17.556533 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550636    2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 22:12:17.556533 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550638    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 22:12:17.556533 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550641    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 22:12:17.556533 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550644    2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 22:12:17.556533 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550646    2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 22:12:17.556533 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550649    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 22:12:17.556533 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550652    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 22:12:17.556533 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550654    2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 22:12:17.557013 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550657    2570 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 22:12:17.557013 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550660    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 22:12:17.557013 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550663    2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 22:12:17.557013 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550665    2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 22:12:17.557013 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550668    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 22:12:17.557013 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550670    2570 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 22:12:17.557013 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550673    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 22:12:17.557013 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550676    2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 22:12:17.557013 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550678    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 22:12:17.557013 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550681    2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 22:12:17.557013 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550683    2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 22:12:17.557013 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550686    2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 22:12:17.557013 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550688    2570 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 22:12:17.557013 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550691    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 22:12:17.557013 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550693    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 22:12:17.557013 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550696    2570 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 22:12:17.557013 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550698    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 22:12:17.557013 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550701    2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 22:12:17.557013 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550703    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 22:12:17.557013 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550706    2570 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 22:12:17.557522 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550709    2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 22:12:17.557522 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550712    2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 22:12:17.557522 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550714    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 22:12:17.557522 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550717    2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 22:12:17.557522 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550719    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 22:12:17.557522 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550722    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 22:12:17.557522 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550725    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 22:12:17.557522 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550727    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 22:12:17.557522 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550729    2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 22:12:17.557522 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550732    2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 22:12:17.557522 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550735    2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 22:12:17.557522 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550737    2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 22:12:17.557522 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550740    2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 22:12:17.557522 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550742    2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 22:12:17.557522 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550745    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 22:12:17.557522 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550748    2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 22:12:17.557522 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550750    2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 22:12:17.557522 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550753    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 22:12:17.557522 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550756    2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 22:12:17.557967 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550758    2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 22:12:17.557967 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550761    2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 22:12:17.557967 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550763    2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 22:12:17.557967 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550766    2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 22:12:17.557967 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550768    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 22:12:17.557967 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550771    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 22:12:17.557967 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550773    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 22:12:17.557967 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550776    2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 22:12:17.557967 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550778    2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 22:12:17.557967 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550781    2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 22:12:17.557967 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550783    2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 22:12:17.557967 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550786    2570 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 22:12:17.557967 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550790    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 22:12:17.557967 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550793    2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 22:12:17.557967 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550796    2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 22:12:17.557967 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550799    2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 22:12:17.557967 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550801    2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 22:12:17.557967 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550804    2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 22:12:17.557967 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550807    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 22:12:17.557967 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550809    2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 22:12:17.558443 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550812    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 22:12:17.558443 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550814    2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 22:12:17.558443 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550817    2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 22:12:17.558443 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550819    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 22:12:17.558443 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550821    2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 22:12:17.558443 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550824    2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 22:12:17.558443 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550826    2570 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 22:12:17.558443 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550829    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 22:12:17.558443 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550832    2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 22:12:17.558443 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550834    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 22:12:17.558443 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550837    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 22:12:17.558443 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550839    2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 22:12:17.558443 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550843    2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 22:12:17.558443 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550846    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 22:12:17.558443 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550849    2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 22:12:17.558443 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550852    2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 22:12:17.558443 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550854    2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 22:12:17.558836 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.550881    2570 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 22:12:17.558836 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550916    2570 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 22:12:17.558836 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550920    2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 22:12:17.558836 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550923    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 22:12:17.558836 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550925    2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 22:12:17.558836 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550928    2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 22:12:17.558836 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550930    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 22:12:17.558836 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550933    2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 22:12:17.558836 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550937    2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 22:12:17.558836 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550940    2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 22:12:17.558836 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550943    2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 22:12:17.558836 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550945    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 22:12:17.558836 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550948    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 22:12:17.558836 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550951    2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 22:12:17.558836 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550954    2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 22:12:17.559193 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550956    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 22:12:17.559193 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550959    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 22:12:17.559193 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550962    2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 22:12:17.559193 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550965    2570 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 22:12:17.559193 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550967    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 22:12:17.559193 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550970    2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 22:12:17.559193 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550972    2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 22:12:17.559193 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550975    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 22:12:17.559193 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550977    2570 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 22:12:17.559193 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550980    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 22:12:17.559193 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550982    2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 22:12:17.559193 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550985    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 22:12:17.559193 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550988    2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 22:12:17.559193 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550990    2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 22:12:17.559193 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550993    2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 22:12:17.559193 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550995    2570 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 22:12:17.559193 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.550998    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 22:12:17.559193 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.551000    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 22:12:17.559193 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.551003    2570 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 22:12:17.559193 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.551005    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 22:12:17.559664 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.551008    2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 22:12:17.559664 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.551012    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 22:12:17.559664 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.551016    2570 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 22:12:17.559664 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.551020    2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 22:12:17.559664 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.551023    2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 22:12:17.559664 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.551029    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 22:12:17.559664 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.551032    2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 22:12:17.559664 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.551036    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 22:12:17.559664 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.551040    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 22:12:17.559664 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.551044    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 22:12:17.559664 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.551048    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 22:12:17.559664 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.551052    2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 22:12:17.559664 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.551056    2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 22:12:17.559664 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.551059    2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 22:12:17.559664 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.551062    2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 22:12:17.559664 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.551064    2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 22:12:17.559664 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.551067    2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 22:12:17.559664 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.551070    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 22:12:17.559664 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.551073    2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 22:12:17.560106 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.551076    2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 22:12:17.560106 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.551079    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 22:12:17.560106 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.551082    2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 22:12:17.560106 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.551085    2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 22:12:17.560106 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.551088    2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 22:12:17.560106 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.551091    2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 22:12:17.560106 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.551093    2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 22:12:17.560106 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.551096    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 22:12:17.560106 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.551099    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 22:12:17.560106 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.551101    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 22:12:17.560106 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.551104    2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 22:12:17.560106 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.551107    2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 22:12:17.560106 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.551109    2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 22:12:17.560106 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.551112    2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 22:12:17.560106 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.551115    2570 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 22:12:17.560106 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.551117    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 22:12:17.560106 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.551120    2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 22:12:17.560106 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.551122    2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 22:12:17.560106 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.551126    2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 22:12:17.560106 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.551129    2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 22:12:17.560645 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.551132    2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 22:12:17.560645 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.551135    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 22:12:17.560645 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.551139    2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 22:12:17.560645 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.551141    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 22:12:17.560645 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.551144    2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 22:12:17.560645 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.551147    2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 22:12:17.560645 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.551150    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 22:12:17.560645 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.551152    2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 22:12:17.560645 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.551154    2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 22:12:17.560645 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.551157    2570 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 22:12:17.560645 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.551159    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 22:12:17.560645 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.551162    2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 22:12:17.560645 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.551166    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 22:12:17.560645 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.551168    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 22:12:17.560645 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.551171    2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 22:12:17.560645 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.551175    2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 22:12:17.560645 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.551179    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 22:12:17.560645 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.551182    2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 22:12:17.560645 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.551184    2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 22:12:17.561125 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.551187    2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 22:12:17.561125 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.551190    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 22:12:17.561125 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.551193    2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 22:12:17.561125 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.551196    2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 22:12:17.561125 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.551198    2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 22:12:17.561125 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.551200    2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 22:12:17.561125 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.551203    2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 22:12:17.561125 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.551206    2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 22:12:17.561125 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.551210    2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 22:12:17.561125 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.551228    2570 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 22:12:17.561125 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.551231    2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 22:12:17.561125 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.551236    2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 22:12:17.561125 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.551239    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 22:12:17.561125 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.551241    2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 22:12:17.561125 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.557638    2570 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 22:12:17.561125 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.557653    2570 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 22:12:17.561125 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.558246    2570 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 22:12:17.561125 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.558252    2570 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 22:12:17.561125 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558309    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 22:12:17.561591 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558314    2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 22:12:17.561591 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558319    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 22:12:17.561591 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558322    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 22:12:17.561591 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558324    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 22:12:17.561591 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558327    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 22:12:17.561591 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558329    2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 22:12:17.561591 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558332    2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 22:12:17.561591 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558336    2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 22:12:17.561591 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558339    2570 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 22:12:17.561591 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558341    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 22:12:17.561591 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558344    2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 22:12:17.561591 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558346    2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 22:12:17.561591 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558349    2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 22:12:17.561591 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558352    2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 22:12:17.561591 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558354    2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 22:12:17.561591 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558357    2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 22:12:17.561591 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558360    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 22:12:17.561591 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558363    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 22:12:17.561591 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558366    2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 22:12:17.561591 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558368    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 22:12:17.562063 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558371    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 22:12:17.562063 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558374    2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 22:12:17.562063 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558376    2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 22:12:17.562063 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558379    2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 22:12:17.562063 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558382    2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 22:12:17.562063 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558384    2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 22:12:17.562063 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558387    2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 22:12:17.562063 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558389    2570 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 22:12:17.562063 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558392    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 22:12:17.562063 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558396    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 22:12:17.562063 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558398    2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 22:12:17.562063 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558401    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 22:12:17.562063 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558404    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 22:12:17.562063 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558406    2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 22:12:17.562063 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558409    2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 22:12:17.562063 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558411    2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 22:12:17.562063 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558414    2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 22:12:17.562063 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558416    2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 22:12:17.562063 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558419    2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 22:12:17.562550 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558421    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 22:12:17.562550 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558424    2570 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 22:12:17.562550 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558426    2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 22:12:17.562550 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558429    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 22:12:17.562550 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558431    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 22:12:17.562550 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558434    2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 22:12:17.562550 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558437    2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 22:12:17.562550 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558439    2570 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 22:12:17.562550 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558442    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 22:12:17.562550 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558445    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 22:12:17.562550 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558448    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 22:12:17.562550 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558450    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 22:12:17.562550 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558453    2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 22:12:17.562550 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558456    2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 22:12:17.562550 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558458    2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 22:12:17.562550 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558461    2570 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 22:12:17.562550 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558463    2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 22:12:17.562550 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558466    2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 22:12:17.562550 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558469    2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 22:12:17.562550 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558471    2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 22:12:17.563014 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558474    2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 22:12:17.563014 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558478    2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 22:12:17.563014 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558480    2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 22:12:17.563014 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558483    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 22:12:17.563014 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558486    2570 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 22:12:17.563014 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558488    2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 22:12:17.563014 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558491    2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 22:12:17.563014 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558495    2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 22:12:17.563014 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558499    2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 22:12:17.563014 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558501    2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 22:12:17.563014 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558505    2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 22:12:17.563014 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558507    2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 22:12:17.563014 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558510    2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 22:12:17.563014 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558512    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 22:12:17.563014 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558515    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 22:12:17.563014 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558517    2570 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 22:12:17.563014 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558520    2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 22:12:17.563014 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558522    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 22:12:17.563014 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558525    2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 22:12:17.563014 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558527    2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 22:12:17.563493 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558531    2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 22:12:17.563493 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558535    2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 22:12:17.563493 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558538    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 22:12:17.563493 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558540    2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 22:12:17.563493 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558543    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 22:12:17.563493 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558545    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 22:12:17.563493 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558548    2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 22:12:17.563493 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558550    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 22:12:17.563493 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558553    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 22:12:17.563493 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558555    2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 22:12:17.563493 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558558    2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 22:12:17.563493 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558560    2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 22:12:17.563493 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558563    2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 22:12:17.563493 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558566    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 22:12:17.563493 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558569    2570 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 22:12:17.563493 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558571    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 22:12:17.563493 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558575    2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 22:12:17.563493 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558578    2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 22:12:17.563493 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558580    2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 22:12:17.563932 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558583    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 22:12:17.563932 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558586    2570 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 22:12:17.563932 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558588    2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 22:12:17.563932 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558591    2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 22:12:17.563932 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558593    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 22:12:17.563932 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558596    2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 22:12:17.563932 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558598    2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 22:12:17.563932 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.558627    2570 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 22:12:17.563932 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558762    2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 22:12:17.563932 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558767    2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 22:12:17.563932 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558769    2570 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 22:12:17.563932 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558773    2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 22:12:17.563932 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558778    2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 22:12:17.563932 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558781    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 22:12:17.563932 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558784    2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 22:12:17.563932 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558786    2570 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 22:12:17.564427 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558789    2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 22:12:17.564427 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558791    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 22:12:17.564427 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558794    2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 22:12:17.564427 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558796    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 22:12:17.564427 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558799    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 22:12:17.564427 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558801    2570 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 22:12:17.564427 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558804    2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 22:12:17.564427 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558807    2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 22:12:17.564427 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558810    2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 22:12:17.564427 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558812    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 22:12:17.564427 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558815    2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 22:12:17.564427 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558818    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 22:12:17.564427 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558820    2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 22:12:17.564427 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558823    2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 22:12:17.564427 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558826    2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 22:12:17.564427 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558828    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 22:12:17.564427 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558831    2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 22:12:17.564427 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558833    2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 22:12:17.564427 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558836    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 22:12:17.564427 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558838    2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 22:12:17.564887 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558841    2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 22:12:17.564887 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558843    2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 22:12:17.564887 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558846    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 22:12:17.564887 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558849    2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 22:12:17.564887 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558851    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 22:12:17.564887 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558854    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 22:12:17.564887 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558856    2570 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 22:12:17.564887 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558859    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 22:12:17.564887 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558861    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 22:12:17.564887 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558864    2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 22:12:17.564887 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558866    2570 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 22:12:17.564887 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558869    2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 22:12:17.564887 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558871    2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 22:12:17.564887 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558874    2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 22:12:17.564887 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558876    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 22:12:17.564887 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558879    2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 22:12:17.564887 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558881    2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 22:12:17.564887 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558884    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 22:12:17.564887 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558887    2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 22:12:17.564887 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558889    2570 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 22:12:17.565372 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558892    2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 22:12:17.565372 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558894    2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 22:12:17.565372 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558897    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 22:12:17.565372 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558900    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 22:12:17.565372 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558902    2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 22:12:17.565372 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558905    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 22:12:17.565372 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558907    2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 22:12:17.565372 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558910    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 22:12:17.565372 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558913    2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 22:12:17.565372 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558916    2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 22:12:17.565372 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558918    2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 22:12:17.565372 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558921    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 22:12:17.565372 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558923    2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 22:12:17.565372 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558926    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 22:12:17.565372 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558928    2570 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 22:12:17.565372 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558931    2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 22:12:17.565372 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558933    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 22:12:17.565372 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558936    2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 22:12:17.565372 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558938    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 22:12:17.565850 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558941    2570 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 22:12:17.565850 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558943    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 22:12:17.565850 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558946    2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 22:12:17.565850 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558948    2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 22:12:17.565850 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558951    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 22:12:17.565850 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558953    2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 22:12:17.565850 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558956    2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 22:12:17.565850 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558959    2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 22:12:17.565850 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558961    2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 22:12:17.565850 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558963    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 22:12:17.565850 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558966    2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 22:12:17.565850 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558969    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 22:12:17.565850 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558971    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 22:12:17.565850 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558974    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 22:12:17.565850 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558978    2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 22:12:17.565850 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558982    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 22:12:17.565850 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558985    2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 22:12:17.565850 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558987    2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 22:12:17.566314 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558990    2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 22:12:17.566314 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558992    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 22:12:17.566314 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558995    2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 22:12:17.566314 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.558998    2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 22:12:17.566314 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559000    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 22:12:17.566314 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559003    2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 22:12:17.566314 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559005    2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 22:12:17.566314 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559008    2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 22:12:17.566314 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559010    2570 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 22:12:17.566314 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559013    2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 22:12:17.566314 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559015    2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 22:12:17.566314 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559018    2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 22:12:17.566314 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559020    2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 22:12:17.566314 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559023    2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 22:12:17.566314 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559025    2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 22:12:17.566314 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559027    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 22:12:17.566314 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559030    2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 22:12:17.566314 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559032    2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 22:12:17.566314 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559034    2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 22:12:17.566314 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559037    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 22:12:17.566781 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559039    2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 22:12:17.566781 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.559062    2570 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 22:12:17.566781 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559096    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 22:12:17.566781 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559100    2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 22:12:17.566781 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559103    2570 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 22:12:17.566781 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559106    2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 22:12:17.566781 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559108    2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 22:12:17.566781 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559111    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 22:12:17.566781 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559114    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 22:12:17.566781 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559116    2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 22:12:17.566781 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559119    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 22:12:17.566781 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559121    2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 22:12:17.566781 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559124    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 22:12:17.566781 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559127    2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 22:12:17.566781 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559129    2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 22:12:17.566781 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559132    2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 22:12:17.567146 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559135    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 22:12:17.567146 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559137    2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 22:12:17.567146 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559140    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 22:12:17.567146 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559142    2570 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 22:12:17.567146 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559145    2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 22:12:17.567146 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559147    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 22:12:17.567146 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559150    2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 22:12:17.567146 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559153    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 22:12:17.567146 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559156    2570 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 22:12:17.567146 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559158    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 22:12:17.567146 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559161    2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 22:12:17.567146 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559163    2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 22:12:17.567146 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559165    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 22:12:17.567146 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559168    2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 22:12:17.567146 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559170    2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 22:12:17.567146 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559173    2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 22:12:17.567146 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559175    2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 22:12:17.567146 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559178    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 22:12:17.567146 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559181    2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 22:12:17.567597 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559183    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 22:12:17.567597 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559186    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 22:12:17.567597 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559188    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 22:12:17.567597 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559192    2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 22:12:17.567597 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559195    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 22:12:17.567597 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559199    2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 22:12:17.567597 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559202    2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 22:12:17.567597 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559204    2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 22:12:17.567597 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559207    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 22:12:17.567597 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559210    2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 22:12:17.567597 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559226    2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 22:12:17.567597 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559230    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 22:12:17.567597 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559234    2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 22:12:17.567597 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559238    2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 22:12:17.567597 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559244    2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 22:12:17.567597 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559247    2570 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 22:12:17.567597 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559250    2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 22:12:17.567597 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559253    2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 22:12:17.567597 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559255    2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 22:12:17.568084 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559258    2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 22:12:17.568084 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559260    2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 22:12:17.568084 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559263    2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 22:12:17.568084 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559265    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 22:12:17.568084 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559268    2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 22:12:17.568084 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559270    2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 22:12:17.568084 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559273    2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 22:12:17.568084 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559275    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 22:12:17.568084 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559277    2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 22:12:17.568084 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559280    2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 22:12:17.568084 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559282    2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 22:12:17.568084 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559285    2570 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 22:12:17.568084 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559289    2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 22:12:17.568084 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559293    2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 22:12:17.568084 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559296    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 22:12:17.568084 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559298    2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 22:12:17.568084 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559302    2570 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 22:12:17.568084 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559305    2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 22:12:17.568084 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559308    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 22:12:17.568084 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559311    2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 22:12:17.568572 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559314    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 22:12:17.568572 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559316    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 22:12:17.568572 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559319    2570 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 22:12:17.568572 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559322    2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 22:12:17.568572 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559324    2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 22:12:17.568572 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559327    2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 22:12:17.568572 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559329    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 22:12:17.568572 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559332    2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 22:12:17.568572 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559334    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 22:12:17.568572 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559337    2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 22:12:17.568572 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559339    2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 22:12:17.568572 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559342    2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 22:12:17.568572 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559344    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 22:12:17.568572 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559347    2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 22:12:17.568572 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559349    2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 22:12:17.568572 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559352    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 22:12:17.568572 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559354    2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 22:12:17.568572 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559357    2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 22:12:17.568572 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559359    2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 22:12:17.568572 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559362    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 22:12:17.569057 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559364    2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 22:12:17.569057 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559367    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 22:12:17.569057 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559369    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 22:12:17.569057 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559372    2570 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 22:12:17.569057 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559374    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 22:12:17.569057 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559377    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 22:12:17.569057 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559380    2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 22:12:17.569057 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559382    2570 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 22:12:17.569057 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559385    2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 22:12:17.569057 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559388    2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 22:12:17.569057 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559391    2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 22:12:17.569057 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559393    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 22:12:17.569057 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559396    2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 22:12:17.569057 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:17.559398    2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 22:12:17.569057 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.559450    2570 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 22:12:17.569057 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.561912    2570 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 03 22:12:17.569057 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.562808    2570 server.go:1014] "Starting client certificate rotation"
Nov 03 22:12:17.569057 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.562903    2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 22:12:17.569057 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.562934    2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 22:12:17.569674 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.564353    2570 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 22:12:17.569674 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.565481    2570 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 22:12:17.575819 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.575803    2570 log.go:25] "Validated CRI v1 runtime API"
Nov 03 22:12:17.580593 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.580569    2570 log.go:25] "Validated CRI v1 image API"
Nov 03 22:12:17.581409 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.581383    2570 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 22:12:17.586534 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.586500    2570 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 e501a563-9a27-497f-9c02-7f2fa7b068d2:/dev/nvme0n1p3 ecdc7f09-2dd5-497e-9bdc-adf35550ec88:/dev/nvme0n1p4]
Nov 03 22:12:17.586616 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.586532    2570 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 03 22:12:17.592058 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.591942    2570 manager.go:217] Machine: {Timestamp:2026-11-03 22:12:17.590676891 +0000 UTC m=+0.301740318 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099416 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec206bb765ff4f0b7df103be4977e5fa SystemUUID:ec206bb7-65ff-4f0b-7df1-03be4977e5fa BootID:72dce58a-b349-428f-9b57-2111e098e01c Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:de:44:3a:dd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:de:44:3a:dd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:f6:9a:4d:ff:93:cf Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 22:12:17.592058 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.592057    2570 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 22:12:17.592168 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.592143    2570 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 22:12:17.594840 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.594810    2570 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 22:12:17.595000 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.594841    2570 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-73.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 22:12:17.595041 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.595003    2570 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 22:12:17.595041 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.595011    2570 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 22:12:17.595091 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.595050    2570 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 22:12:17.595091 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.595068    2570 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 22:12:17.596307 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.596296    2570 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 22:12:17.599797 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.599785    2570 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 22:12:17.599913 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.599904    2570 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 22:12:17.600626 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.600615    2570 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 22:12:17.600658 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.600631    2570 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 22:12:17.600658 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.600642    2570 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 22:12:17.600658 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.600650    2570 kubelet.go:405] "Adding apiserver pod source"
Nov 03 22:12:17.600749 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.600659    2570 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 22:12:17.601027 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.601015    2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 22:12:17.602782 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.602759    2570 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 22:12:17.608113 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.608100    2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 22:12:17.608780 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.608768    2570 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 22:12:17.608818 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.608798    2570 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 22:12:17.608950 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.608937    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 22:12:17.608950 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.608952    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 22:12:17.609060 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.608960    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 22:12:17.609060 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.608965    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 22:12:17.609060 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.608970    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 22:12:17.609060 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.608976    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 22:12:17.609491 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.609481    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 22:12:17.609491 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.609491    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 22:12:17.609548 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.609498    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 22:12:17.609548 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.609504    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 22:12:17.609548 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.609514    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 22:12:17.609548 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.609523    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 22:12:17.610140 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.610127    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 22:12:17.610184 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.610144    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 22:12:17.613665 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.613651    2570 server.go:1263] "Started kubelet"
Nov 03 22:12:17.613763 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.613724    2570 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 22:12:17.613866 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.613813    2570 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 22:12:17.613911 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.613898    2570 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 22:12:17.615074 ip-10-0-139-73 systemd[1]: Started Kubernetes Kubelet.
Nov 03 22:12:17.616025 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.615414    2570 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 22:12:17.617680 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.616866    2570 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 22:12:17.623328 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:17.621343    2570 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 03 22:12:17.623328 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.622658    2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 22:12:17.623510 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.623373    2570 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 22:12:17.623510 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:17.623438    2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-73.ec2.internal\" not found"
Nov 03 22:12:17.624296 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.624278    2570 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 22:12:17.624296 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.624295    2570 factory.go:55] Registering systemd factory
Nov 03 22:12:17.624296 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.624301    2570 factory.go:223] Registration of the systemd container factory successfully
Nov 03 22:12:17.624460 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.624312    2570 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 22:12:17.624460 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.624316    2570 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 22:12:17.624460 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.624333    2570 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 22:12:17.624460 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.624381    2570 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 22:12:17.624460 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.624387    2570 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 22:12:17.624657 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.624548    2570 factory.go:153] Registering CRI-O factory
Nov 03 22:12:17.624657 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.624560    2570 factory.go:223] Registration of the crio container factory successfully
Nov 03 22:12:17.624657 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.624573    2570 factory.go:103] Registering Raw factory
Nov 03 22:12:17.624657 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.624588    2570 manager.go:1199] Started watching for new ooms in manager
Nov 03 22:12:17.625534 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.625520    2570 manager.go:319] Starting recovery of all containers
Nov 03 22:12:17.635345 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.635331    2570 manager.go:324] Recovery completed
Nov 03 22:12:17.639693 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.639679    2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 22:12:17.641477 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.641460    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-73.ec2.internal" event="NodeHasSufficientMemory"
Nov 03 22:12:17.641567 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.641495    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-73.ec2.internal" event="NodeHasNoDiskPressure"
Nov 03 22:12:17.641567 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.641511    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-73.ec2.internal" event="NodeHasSufficientPID"
Nov 03 22:12:17.642128 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.642113    2570 cpu_manager.go:226] "Starting" policy="none"
Nov 03 22:12:17.642203 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.642127    2570 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 22:12:17.642203 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.642147    2570 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 22:12:17.642203 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.642170    2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 22:12:17.642950 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.642936    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-73.ec2.internal" event="NodeHasSufficientMemory"
Nov 03 22:12:17.643010 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.642963    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-73.ec2.internal" event="NodeHasNoDiskPressure"
Nov 03 22:12:17.643010 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.642974    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-73.ec2.internal" event="NodeHasSufficientPID"
Nov 03 22:12:17.644382 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.644364    2570 policy_none.go:50] "Start"
Nov 03 22:12:17.644449 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.644386    2570 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 22:12:17.644449 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.644401    2570 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 22:12:17.646666 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.646647    2570 policy_none.go:44] "Start"
Nov 03 22:12:17.648355 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:17.648330    2570 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-73.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 03 22:12:17.648355 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.648341    2570 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-73.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 03 22:12:17.659086 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:17.658350    2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-73.ec2.internal.18e4a89b8b7871ac  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-73.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-73.ec2.internal,},FirstTimestamp:2026-11-03 22:12:17.613623724 +0000 UTC m=+0.324687151,LastTimestamp:2026-11-03 22:12:17.613623724 +0000 UTC m=+0.324687151,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-73.ec2.internal,}"
Nov 03 22:12:17.662626 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.662607    2570 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 22:12:17.662721 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.662647    2570 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-q29v6"
Nov 03 22:12:17.668523 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:17.668498    2570 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Nov 03 22:12:17.668644 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:17.668588    2570 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Nov 03 22:12:17.669203 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.669190    2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-q29v6"
Nov 03 22:12:17.679871 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.679838    2570 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 22:12:17.684348 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.684329    2570 manager.go:341] "Starting Device Plugin manager"
Nov 03 22:12:17.684426 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:17.684354    2570 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 22:12:17.684426 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.684363    2570 server.go:87] "Starting device plugin registration server"
Nov 03 22:12:17.684678 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.684664    2570 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 22:12:17.684720 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.684681    2570 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 22:12:17.684762 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.684735    2570 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 22:12:17.684859 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.684820    2570 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 22:12:17.684859 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.684841    2570 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 22:12:17.685465 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:17.685447    2570 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 22:12:17.685554 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:17.685493    2570 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-73.ec2.internal\" not found"
Nov 03 22:12:17.692090 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.692074    2570 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 22:12:17.748402 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.748373    2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 22:12:17.749574 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.749560    2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 22:12:17.749646 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.749579    2570 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 22:12:17.749646 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.749608    2570 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 22:12:17.749727 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:17.749646    2570 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Nov 03 22:12:17.762251 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.762232    2570 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 22:12:17.785589 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.785566    2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 22:12:17.786460 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.786442    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-73.ec2.internal" event="NodeHasSufficientMemory"
Nov 03 22:12:17.786561 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.786476    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-73.ec2.internal" event="NodeHasNoDiskPressure"
Nov 03 22:12:17.786561 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.786490    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-73.ec2.internal" event="NodeHasSufficientPID"
Nov 03 22:12:17.786561 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.786526    2570 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-73.ec2.internal"
Nov 03 22:12:17.804001 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.803942    2570 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-139-73.ec2.internal"
Nov 03 22:12:17.804001 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:17.803971    2570 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-73.ec2.internal\": node \"ip-10-0-139-73.ec2.internal\" not found"
Nov 03 22:12:17.850727 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.850556    2570 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-139-73.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-73.ec2.internal"]
Nov 03 22:12:17.853045 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.853030    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-73.ec2.internal"
Nov 03 22:12:17.854896 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.854878    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-73.ec2.internal"
Nov 03 22:12:17.872696 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.872670    2570 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-73.ec2.internal"
Nov 03 22:12:17.878821 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.878803    2570 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-73.ec2.internal"
Nov 03 22:12:17.901044 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.901024    2570 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 22:12:17.924809 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.924790    2570 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 22:12:17.924878 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.924836    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a7264ab43e0f926a44edf6ae184aafb2-config\") pod \"kube-apiserver-proxy-ip-10-0-139-73.ec2.internal\" (UID: \"a7264ab43e0f926a44edf6ae184aafb2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-73.ec2.internal"
Nov 03 22:12:17.924878 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.924859    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9e2325705fcd5fefcf1f90504938970b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-73.ec2.internal\" (UID: \"9e2325705fcd5fefcf1f90504938970b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-73.ec2.internal"
Nov 03 22:12:17.924878 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:17.924876    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e2325705fcd5fefcf1f90504938970b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-73.ec2.internal\" (UID: \"9e2325705fcd5fefcf1f90504938970b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-73.ec2.internal"
Nov 03 22:12:18.025345 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.025311    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e2325705fcd5fefcf1f90504938970b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-73.ec2.internal\" (UID: \"9e2325705fcd5fefcf1f90504938970b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-73.ec2.internal"
Nov 03 22:12:18.025345 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.025348    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a7264ab43e0f926a44edf6ae184aafb2-config\") pod \"kube-apiserver-proxy-ip-10-0-139-73.ec2.internal\" (UID: \"a7264ab43e0f926a44edf6ae184aafb2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-73.ec2.internal"
Nov 03 22:12:18.025550 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.025371    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9e2325705fcd5fefcf1f90504938970b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-73.ec2.internal\" (UID: \"9e2325705fcd5fefcf1f90504938970b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-73.ec2.internal"
Nov 03 22:12:18.025550 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.025428    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9e2325705fcd5fefcf1f90504938970b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-73.ec2.internal\" (UID: \"9e2325705fcd5fefcf1f90504938970b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-73.ec2.internal"
Nov 03 22:12:18.025550 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.025428    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a7264ab43e0f926a44edf6ae184aafb2-config\") pod \"kube-apiserver-proxy-ip-10-0-139-73.ec2.internal\" (UID: \"a7264ab43e0f926a44edf6ae184aafb2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-73.ec2.internal"
Nov 03 22:12:18.025550 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.025429    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e2325705fcd5fefcf1f90504938970b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-73.ec2.internal\" (UID: \"9e2325705fcd5fefcf1f90504938970b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-73.ec2.internal"
Nov 03 22:12:18.206038 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.205951    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-73.ec2.internal"
Nov 03 22:12:18.229048 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.229027    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-73.ec2.internal"
Nov 03 22:12:18.563073 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.563020    2570 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 03 22:12:18.601433 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.601404    2570 apiserver.go:52] "Watching apiserver"
Nov 03 22:12:18.623311 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.623272    2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 03 22:12:18.640477 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.640426    2570 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 22:12:18.641679 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.641618    2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-jn5wf","openshift-dns/node-resolver-x7cpm","openshift-image-registry/node-ca-p4p25","openshift-ovn-kubernetes/ovnkube-node-mx5q8","kube-system/global-pull-secret-syncer-66m8v","kube-system/konnectivity-agent-7p8pl","openshift-cluster-node-tuning-operator/tuned-9ltv9","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-73.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zcr2","openshift-network-diagnostics/network-check-target-82h4r","openshift-network-operator/iptables-alerter-7rqgq","kube-system/kube-apiserver-proxy-ip-10-0-139-73.ec2.internal","openshift-multus/multus-2k9fs","openshift-multus/multus-additional-cni-plugins-4sm9h"]
Nov 03 22:12:18.644640 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.644619    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jn5wf"
Nov 03 22:12:18.644729 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:18.644702    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jn5wf" podUID="9b071df2-94c8-4459-89f2-f4b448ceb581"
Nov 03 22:12:18.647377 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.647357    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-x7cpm"
Nov 03 22:12:18.649884 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.649858    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-p4p25"
Nov 03 22:12:18.652354 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.652329    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:18.654631 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.654611    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-66m8v"
Nov 03 22:12:18.654726 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:18.654676    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-66m8v" podUID="8ea0b360-1d72-4d40-935e-1394fecad74c"
Nov 03 22:12:18.661918 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.661897    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7p8pl"
Nov 03 22:12:18.662457 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.662432    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 22:12:18.662545 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.662487    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 22:12:18.662602 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.662585    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-jwdqh\""
Nov 03 22:12:18.663417 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.663394    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 22:12:18.663519 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.663421    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 22:12:18.663519 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.663463    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 22:12:18.663519 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.663476    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-mdchn\""
Nov 03 22:12:18.664465 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.664446    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9ltv9"
Nov 03 22:12:18.665957 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.665937    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 22:12:18.666259 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.666242    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 22:12:18.666344 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.666277    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 22:12:18.666446 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.666432    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 22:12:18.666495 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.666444    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 22:12:18.666495 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.666449    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-wf9zk\""
Nov 03 22:12:18.666559 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.666521    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 22:12:18.666870 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.666852    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zcr2"
Nov 03 22:12:18.667259 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.667239    2570 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 22:12:18.669341 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.669322    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-82h4r"
Nov 03 22:12:18.669413 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:18.669392    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-82h4r" podUID="1b792dc0-5e1c-40c4-a6c6-f68b4c6c8aac"
Nov 03 22:12:18.671862 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.671846    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7rqgq"
Nov 03 22:12:18.674379 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.674363    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:18.674459 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.674281    2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 22:07:17 +0000 UTC" deadline="2028-03-29 13:23:38.90297183 +0000 UTC"
Nov 03 22:12:18.674459 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.674400    2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12279h11m20.228575984s"
Nov 03 22:12:18.676422 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.676404    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 22:12:18.676830 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.676816    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4sm9h"
Nov 03 22:12:18.697678 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.697654    2570 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-tlpbf"
Nov 03 22:12:18.708689 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.708666    2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-tlpbf"
Nov 03 22:12:18.713508 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:18.713478    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e2325705fcd5fefcf1f90504938970b.slice/crio-05e93303e59fe467c6f7d583564effdd8a09180902c8db08e8211ba9ad1b3861 WatchSource:0}: Error finding container 05e93303e59fe467c6f7d583564effdd8a09180902c8db08e8211ba9ad1b3861: Status 404 returned error can't find the container with id 05e93303e59fe467c6f7d583564effdd8a09180902c8db08e8211ba9ad1b3861
Nov 03 22:12:18.713805 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:18.713787    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7264ab43e0f926a44edf6ae184aafb2.slice/crio-972ef7c557baa631f2303101cdfb55622fd8b8ceea4288d2f527e38c4bd4dfa8 WatchSource:0}: Error finding container 972ef7c557baa631f2303101cdfb55622fd8b8ceea4288d2f527e38c4bd4dfa8: Status 404 returned error can't find the container with id 972ef7c557baa631f2303101cdfb55622fd8b8ceea4288d2f527e38c4bd4dfa8
Nov 03 22:12:18.721162 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.721030    2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 22:12:18.725591 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.725557    2570 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 22:12:18.727113 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.727086    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ssqq\" (UniqueName: \"kubernetes.io/projected/cc66a3de-7e7e-4855-8f17-db97dcfa815a-kube-api-access-2ssqq\") pod \"node-ca-p4p25\" (UID: \"cc66a3de-7e7e-4855-8f17-db97dcfa815a\") " pod="openshift-image-registry/node-ca-p4p25"
Nov 03 22:12:18.727196 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.727130    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-host-slash\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:18.727238 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.727190    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/94868020-2e98-4295-8db5-043461cc6e9d-hosts-file\") pod \"node-resolver-x7cpm\" (UID: \"94868020-2e98-4295-8db5-043461cc6e9d\") " pod="openshift-dns/node-resolver-x7cpm"
Nov 03 22:12:18.727273 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.727239    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fa1fd639-c265-4048-b9be-a583885ccb4e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8zcr2\" (UID: \"fa1fd639-c265-4048-b9be-a583885ccb4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zcr2"
Nov 03 22:12:18.727273 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.727258    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fa1fd639-c265-4048-b9be-a583885ccb4e-sys-fs\") pod \"aws-ebs-csi-driver-node-8zcr2\" (UID: \"fa1fd639-c265-4048-b9be-a583885ccb4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zcr2"
Nov 03 22:12:18.727413 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.727299    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/089f9a4b-8eec-4758-811b-e506b41cecbb-system-cni-dir\") pod \"multus-additional-cni-plugins-4sm9h\" (UID: \"089f9a4b-8eec-4758-811b-e506b41cecbb\") " pod="openshift-multus/multus-additional-cni-plugins-4sm9h"
Nov 03 22:12:18.727413 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.727318    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/089f9a4b-8eec-4758-811b-e506b41cecbb-cnibin\") pod \"multus-additional-cni-plugins-4sm9h\" (UID: \"089f9a4b-8eec-4758-811b-e506b41cecbb\") " pod="openshift-multus/multus-additional-cni-plugins-4sm9h"
Nov 03 22:12:18.727413 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.727377    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/089f9a4b-8eec-4758-811b-e506b41cecbb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4sm9h\" (UID: \"089f9a4b-8eec-4758-811b-e506b41cecbb\") " pod="openshift-multus/multus-additional-cni-plugins-4sm9h"
Nov 03 22:12:18.727529 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.727439    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-host-cni-netd\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:18.727529 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.727483    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a06ec3e9-0a6e-46d8-ade5-13d657a6284c-lib-modules\") pod \"tuned-9ltv9\" (UID: \"a06ec3e9-0a6e-46d8-ade5-13d657a6284c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ltv9"
Nov 03 22:12:18.727529 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.727520    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/867a2a76-8aa3-4b07-98bf-913c2d2a5562-iptables-alerter-script\") pod \"iptables-alerter-7rqgq\" (UID: \"867a2a76-8aa3-4b07-98bf-913c2d2a5562\") " pod="openshift-network-operator/iptables-alerter-7rqgq"
Nov 03 22:12:18.727681 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.727538    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b071df2-94c8-4459-89f2-f4b448ceb581-metrics-certs\") pod \"network-metrics-daemon-jn5wf\" (UID: \"9b071df2-94c8-4459-89f2-f4b448ceb581\") " pod="openshift-multus/network-metrics-daemon-jn5wf"
Nov 03 22:12:18.727681 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.727636    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:18.727776 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.727744    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrr2v\" (UniqueName: \"kubernetes.io/projected/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-kube-api-access-nrr2v\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:18.727814 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.727791    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a06ec3e9-0a6e-46d8-ade5-13d657a6284c-etc-systemd\") pod \"tuned-9ltv9\" (UID: \"a06ec3e9-0a6e-46d8-ade5-13d657a6284c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ltv9"
Nov 03 22:12:18.727850 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.727816    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a06ec3e9-0a6e-46d8-ade5-13d657a6284c-sys\") pod \"tuned-9ltv9\" (UID: \"a06ec3e9-0a6e-46d8-ade5-13d657a6284c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ltv9"
Nov 03 22:12:18.727850 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.727841    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-host-run-multus-certs\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:18.727923 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.727867    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shrkm\" (UniqueName: \"kubernetes.io/projected/fa1fd639-c265-4048-b9be-a583885ccb4e-kube-api-access-shrkm\") pod \"aws-ebs-csi-driver-node-8zcr2\" (UID: \"fa1fd639-c265-4048-b9be-a583885ccb4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zcr2"
Nov 03 22:12:18.727923 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.727913    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/089f9a4b-8eec-4758-811b-e506b41cecbb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4sm9h\" (UID: \"089f9a4b-8eec-4758-811b-e506b41cecbb\") " pod="openshift-multus/multus-additional-cni-plugins-4sm9h"
Nov 03 22:12:18.728005 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.727939    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-run-systemd\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:18.728005 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.727954    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/867a2a76-8aa3-4b07-98bf-913c2d2a5562-host-slash\") pod \"iptables-alerter-7rqgq\" (UID: \"867a2a76-8aa3-4b07-98bf-913c2d2a5562\") " pod="openshift-network-operator/iptables-alerter-7rqgq"
Nov 03 22:12:18.728005 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.727972    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-host-var-lib-cni-multus\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:18.728102 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.728031    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-multus-conf-dir\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:18.728102 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.728067    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8ea0b360-1d72-4d40-935e-1394fecad74c-original-pull-secret\") pod \"global-pull-secret-syncer-66m8v\" (UID: \"8ea0b360-1d72-4d40-935e-1394fecad74c\") " pod="kube-system/global-pull-secret-syncer-66m8v"
Nov 03 22:12:18.728102 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.728090    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-run-ovn\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:18.728194 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.728115    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-log-socket\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:18.728194 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.728140    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a06ec3e9-0a6e-46d8-ade5-13d657a6284c-host\") pod \"tuned-9ltv9\" (UID: \"a06ec3e9-0a6e-46d8-ade5-13d657a6284c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ltv9"
Nov 03 22:12:18.728194 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.728162    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a06ec3e9-0a6e-46d8-ade5-13d657a6284c-etc-sysconfig\") pod \"tuned-9ltv9\" (UID: \"a06ec3e9-0a6e-46d8-ade5-13d657a6284c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ltv9"
Nov 03 22:12:18.728347 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.728193    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a06ec3e9-0a6e-46d8-ade5-13d657a6284c-tmp\") pod \"tuned-9ltv9\" (UID: \"a06ec3e9-0a6e-46d8-ade5-13d657a6284c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ltv9"
Nov 03 22:12:18.728347 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.728216    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-system-cni-dir\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:18.728347 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.728274    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fa1fd639-c265-4048-b9be-a583885ccb4e-device-dir\") pod \"aws-ebs-csi-driver-node-8zcr2\" (UID: \"fa1fd639-c265-4048-b9be-a583885ccb4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zcr2"
Nov 03 22:12:18.728347 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.728304    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-host-run-netns\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:18.728538 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.728359    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a06ec3e9-0a6e-46d8-ade5-13d657a6284c-var-lib-kubelet\") pod \"tuned-9ltv9\" (UID: \"a06ec3e9-0a6e-46d8-ade5-13d657a6284c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ltv9"
Nov 03 22:12:18.728538 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.728386    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-cnibin\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:18.728538 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.728443    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fa1fd639-c265-4048-b9be-a583885ccb4e-etc-selinux\") pod \"aws-ebs-csi-driver-node-8zcr2\" (UID: \"fa1fd639-c265-4048-b9be-a583885ccb4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zcr2"
Nov 03 22:12:18.728538 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.728481    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-host-run-ovn-kubernetes\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:18.728538 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.728522    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-env-overrides\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:18.728722 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.728566    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-multus-daemon-config\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:18.728722 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.728604    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-etc-kubernetes\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:18.728722 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.728633    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-ovnkube-script-lib\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:18.728722 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.728658    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a06ec3e9-0a6e-46d8-ade5-13d657a6284c-etc-sysctl-conf\") pod \"tuned-9ltv9\" (UID: \"a06ec3e9-0a6e-46d8-ade5-13d657a6284c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ltv9"
Nov 03 22:12:18.728722 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.728692    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdtpv\" (UniqueName: \"kubernetes.io/projected/1b792dc0-5e1c-40c4-a6c6-f68b4c6c8aac-kube-api-access-qdtpv\") pod \"network-check-target-82h4r\" (UID: \"1b792dc0-5e1c-40c4-a6c6-f68b4c6c8aac\") " pod="openshift-network-diagnostics/network-check-target-82h4r"
Nov 03 22:12:18.728908 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.728726    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-host-var-lib-cni-bin\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:18.728908 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.728791    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhr4l\" (UniqueName: \"kubernetes.io/projected/9b071df2-94c8-4459-89f2-f4b448ceb581-kube-api-access-mhr4l\") pod \"network-metrics-daemon-jn5wf\" (UID: \"9b071df2-94c8-4459-89f2-f4b448ceb581\") " pod="openshift-multus/network-metrics-daemon-jn5wf"
Nov 03 22:12:18.728908 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.728821    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-run-openvswitch\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:18.728908 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.728838    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a06ec3e9-0a6e-46d8-ade5-13d657a6284c-etc-modprobe-d\") pod \"tuned-9ltv9\" (UID: \"a06ec3e9-0a6e-46d8-ade5-13d657a6284c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ltv9"
Nov 03 22:12:18.728908 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.728858    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-multus-cni-dir\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:18.728908 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.728899    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/089f9a4b-8eec-4758-811b-e506b41cecbb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4sm9h\" (UID: \"089f9a4b-8eec-4758-811b-e506b41cecbb\") " pod="openshift-multus/multus-additional-cni-plugins-4sm9h"
Nov 03 22:12:18.729101 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.728917    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-systemd-units\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:18.729101 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.728940    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-ovn-node-metrics-cert\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:18.729101 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.729008    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-cni-binary-copy\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:18.729101 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.729052    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fswpg\" (UniqueName: \"kubernetes.io/projected/94868020-2e98-4295-8db5-043461cc6e9d-kube-api-access-fswpg\") pod \"node-resolver-x7cpm\" (UID: \"94868020-2e98-4295-8db5-043461cc6e9d\") " pod="openshift-dns/node-resolver-x7cpm"
Nov 03 22:12:18.729101 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.729088    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fa1fd639-c265-4048-b9be-a583885ccb4e-socket-dir\") pod \"aws-ebs-csi-driver-node-8zcr2\" (UID: \"fa1fd639-c265-4048-b9be-a583885ccb4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zcr2"
Nov 03 22:12:18.729248 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.729134    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cc66a3de-7e7e-4855-8f17-db97dcfa815a-serviceca\") pod \"node-ca-p4p25\" (UID: \"cc66a3de-7e7e-4855-8f17-db97dcfa815a\") " pod="openshift-image-registry/node-ca-p4p25"
Nov 03 22:12:18.729248 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.729172    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-host-run-netns\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:18.729248 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.729211    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8187068e-17ec-4e6d-bb8e-1d0312aa6674-agent-certs\") pod \"konnectivity-agent-7p8pl\" (UID: \"8187068e-17ec-4e6d-bb8e-1d0312aa6674\") " pod="kube-system/konnectivity-agent-7p8pl"
Nov 03 22:12:18.729375 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.729250    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8187068e-17ec-4e6d-bb8e-1d0312aa6674-konnectivity-ca\") pod \"konnectivity-agent-7p8pl\" (UID: \"8187068e-17ec-4e6d-bb8e-1d0312aa6674\") " pod="kube-system/konnectivity-agent-7p8pl"
Nov 03 22:12:18.729375 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.729286    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a06ec3e9-0a6e-46d8-ade5-13d657a6284c-run\") pod \"tuned-9ltv9\" (UID: \"a06ec3e9-0a6e-46d8-ade5-13d657a6284c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ltv9"
Nov 03 22:12:18.729375 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.729312    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7l82\" (UniqueName: \"kubernetes.io/projected/a06ec3e9-0a6e-46d8-ade5-13d657a6284c-kube-api-access-t7l82\") pod \"tuned-9ltv9\" (UID: \"a06ec3e9-0a6e-46d8-ade5-13d657a6284c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ltv9"
Nov 03 22:12:18.729375 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.729328    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6x5m\" (UniqueName: \"kubernetes.io/projected/867a2a76-8aa3-4b07-98bf-913c2d2a5562-kube-api-access-f6x5m\") pod \"iptables-alerter-7rqgq\" (UID: \"867a2a76-8aa3-4b07-98bf-913c2d2a5562\") " pod="openshift-network-operator/iptables-alerter-7rqgq"
Nov 03 22:12:18.729507 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.729377    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-hostroot\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:18.729507 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.729418    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/089f9a4b-8eec-4758-811b-e506b41cecbb-cni-binary-copy\") pod \"multus-additional-cni-plugins-4sm9h\" (UID: \"089f9a4b-8eec-4758-811b-e506b41cecbb\") " pod="openshift-multus/multus-additional-cni-plugins-4sm9h"
Nov 03 22:12:18.729507 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.729468    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bjhs\" (UniqueName: \"kubernetes.io/projected/089f9a4b-8eec-4758-811b-e506b41cecbb-kube-api-access-7bjhs\") pod \"multus-additional-cni-plugins-4sm9h\" (UID: \"089f9a4b-8eec-4758-811b-e506b41cecbb\") " pod="openshift-multus/multus-additional-cni-plugins-4sm9h"
Nov 03 22:12:18.729507 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.729486    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-node-log\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:18.729658 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.729508    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a06ec3e9-0a6e-46d8-ade5-13d657a6284c-etc-tuned\") pod \"tuned-9ltv9\" (UID: \"a06ec3e9-0a6e-46d8-ade5-13d657a6284c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ltv9"
Nov 03 22:12:18.729658 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.729527    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-multus-socket-dir-parent\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:18.729658 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.729560    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-png47\" (UniqueName: \"kubernetes.io/projected/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-kube-api-access-png47\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:18.729658 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.729582    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8ea0b360-1d72-4d40-935e-1394fecad74c-kubelet-config\") pod \"global-pull-secret-syncer-66m8v\" (UID: \"8ea0b360-1d72-4d40-935e-1394fecad74c\") " pod="kube-system/global-pull-secret-syncer-66m8v"
Nov 03 22:12:18.729658 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.729638    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a06ec3e9-0a6e-46d8-ade5-13d657a6284c-etc-kubernetes\") pod \"tuned-9ltv9\" (UID: \"a06ec3e9-0a6e-46d8-ade5-13d657a6284c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ltv9"
Nov 03 22:12:18.729880 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.729675    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fa1fd639-c265-4048-b9be-a583885ccb4e-registration-dir\") pod \"aws-ebs-csi-driver-node-8zcr2\" (UID: \"fa1fd639-c265-4048-b9be-a583885ccb4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zcr2"
Nov 03 22:12:18.729880 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.729710    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-host-kubelet\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:18.729880 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.729747    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-var-lib-openvswitch\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:18.729880 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.729778    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-host-cni-bin\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:18.729880 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.729810    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-ovnkube-config\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:18.729880 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.729852    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-host-var-lib-kubelet\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:18.729880 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.729878    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94868020-2e98-4295-8db5-043461cc6e9d-tmp-dir\") pod \"node-resolver-x7cpm\" (UID: \"94868020-2e98-4295-8db5-043461cc6e9d\") " pod="openshift-dns/node-resolver-x7cpm"
Nov 03 22:12:18.730120 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.729896    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cc66a3de-7e7e-4855-8f17-db97dcfa815a-host\") pod \"node-ca-p4p25\" (UID: \"cc66a3de-7e7e-4855-8f17-db97dcfa815a\") " pod="openshift-image-registry/node-ca-p4p25"
Nov 03 22:12:18.730120 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.729961    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-etc-openvswitch\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:18.730120 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.729997    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a06ec3e9-0a6e-46d8-ade5-13d657a6284c-etc-sysctl-d\") pod \"tuned-9ltv9\" (UID: \"a06ec3e9-0a6e-46d8-ade5-13d657a6284c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ltv9"
Nov 03 22:12:18.730120 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.730037    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-host-run-k8s-cni-cncf-io\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:18.730120 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.730073    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8ea0b360-1d72-4d40-935e-1394fecad74c-dbus\") pod \"global-pull-secret-syncer-66m8v\" (UID: \"8ea0b360-1d72-4d40-935e-1394fecad74c\") " pod="kube-system/global-pull-secret-syncer-66m8v"
Nov 03 22:12:18.753593 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.753521    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-73.ec2.internal" event={"ID":"9e2325705fcd5fefcf1f90504938970b","Type":"ContainerStarted","Data":"05e93303e59fe467c6f7d583564effdd8a09180902c8db08e8211ba9ad1b3861"}
Nov 03 22:12:18.754690 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.754668    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-73.ec2.internal" event={"ID":"a7264ab43e0f926a44edf6ae184aafb2","Type":"ContainerStarted","Data":"972ef7c557baa631f2303101cdfb55622fd8b8ceea4288d2f527e38c4bd4dfa8"}
Nov 03 22:12:18.770477 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.770454    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 22:12:18.770477 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.770465    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 22:12:18.770628 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.770477    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 22:12:18.770628 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.770531    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 22:12:18.770734 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.770629    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 22:12:18.770734 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.770650    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 22:12:18.770734 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.770692    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 22:12:18.770734 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.770712    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-h4jdd\""
Nov 03 22:12:18.770900 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.770791    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 22:12:18.770900 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.770796    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 22:12:18.770900 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.770837    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 22:12:18.770900 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.770853    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 22:12:18.771028 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.770908    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 22:12:18.771028 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.770911    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7shc8\""
Nov 03 22:12:18.771028 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.770931    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-wvrhf\""
Nov 03 22:12:18.771028 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.770957    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 22:12:18.771028 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.770971    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 22:12:18.771028 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.771004    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 22:12:18.771028 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.771009    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-nt8s4\""
Nov 03 22:12:18.771273 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.771138    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4tpvw\""
Nov 03 22:12:18.771273 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.771154    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rz46p\""
Nov 03 22:12:18.831154 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.831074    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-host-run-ovn-kubernetes\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:18.831154 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.831103    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-env-overrides\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:18.831154 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.831127    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-multus-daemon-config\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:18.831443 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.831236    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-host-run-ovn-kubernetes\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:18.831443 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.831284    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-etc-kubernetes\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:18.831443 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.831312    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-ovnkube-script-lib\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:18.831443 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.831338    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a06ec3e9-0a6e-46d8-ade5-13d657a6284c-etc-sysctl-conf\") pod \"tuned-9ltv9\" (UID: \"a06ec3e9-0a6e-46d8-ade5-13d657a6284c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ltv9"
Nov 03 22:12:18.831443 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.831362    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdtpv\" (UniqueName: \"kubernetes.io/projected/1b792dc0-5e1c-40c4-a6c6-f68b4c6c8aac-kube-api-access-qdtpv\") pod \"network-check-target-82h4r\" (UID: \"1b792dc0-5e1c-40c4-a6c6-f68b4c6c8aac\") " pod="openshift-network-diagnostics/network-check-target-82h4r"
Nov 03 22:12:18.831443 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.831407    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-host-var-lib-cni-bin\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:18.831443 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.831423    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-etc-kubernetes\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:18.831746 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.831436    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhr4l\" (UniqueName: \"kubernetes.io/projected/9b071df2-94c8-4459-89f2-f4b448ceb581-kube-api-access-mhr4l\") pod \"network-metrics-daemon-jn5wf\" (UID: \"9b071df2-94c8-4459-89f2-f4b448ceb581\") " pod="openshift-multus/network-metrics-daemon-jn5wf"
Nov 03 22:12:18.831746 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.831476    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-host-var-lib-cni-bin\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:18.831746 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.831484    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-run-openvswitch\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:18.831746 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.831516    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a06ec3e9-0a6e-46d8-ade5-13d657a6284c-etc-sysctl-conf\") pod \"tuned-9ltv9\" (UID: \"a06ec3e9-0a6e-46d8-ade5-13d657a6284c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ltv9"
Nov 03 22:12:18.831746 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.831521    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-run-openvswitch\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:18.831746 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.831543    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a06ec3e9-0a6e-46d8-ade5-13d657a6284c-etc-modprobe-d\") pod \"tuned-9ltv9\" (UID: \"a06ec3e9-0a6e-46d8-ade5-13d657a6284c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ltv9"
Nov 03 22:12:18.831746 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.831573    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-multus-cni-dir\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:18.831746 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.831603    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/089f9a4b-8eec-4758-811b-e506b41cecbb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4sm9h\" (UID: \"089f9a4b-8eec-4758-811b-e506b41cecbb\") " pod="openshift-multus/multus-additional-cni-plugins-4sm9h"
Nov 03 22:12:18.831746 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.831632    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-systemd-units\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:18.831746 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.831662    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-ovn-node-metrics-cert\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:18.831746 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.831687    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-cni-binary-copy\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:18.831746 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.831714    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fswpg\" (UniqueName: \"kubernetes.io/projected/94868020-2e98-4295-8db5-043461cc6e9d-kube-api-access-fswpg\") pod \"node-resolver-x7cpm\" (UID: \"94868020-2e98-4295-8db5-043461cc6e9d\") " pod="openshift-dns/node-resolver-x7cpm"
Nov 03 22:12:18.831746 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.831730    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a06ec3e9-0a6e-46d8-ade5-13d657a6284c-etc-modprobe-d\") pod \"tuned-9ltv9\" (UID: \"a06ec3e9-0a6e-46d8-ade5-13d657a6284c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ltv9"
Nov 03 22:12:18.831746 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.831740    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fa1fd639-c265-4048-b9be-a583885ccb4e-socket-dir\") pod \"aws-ebs-csi-driver-node-8zcr2\" (UID: \"fa1fd639-c265-4048-b9be-a583885ccb4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zcr2"
Nov 03 22:12:18.832240 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.831769    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cc66a3de-7e7e-4855-8f17-db97dcfa815a-serviceca\") pod \"node-ca-p4p25\" (UID: \"cc66a3de-7e7e-4855-8f17-db97dcfa815a\") " pod="openshift-image-registry/node-ca-p4p25"
Nov 03 22:12:18.832240 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.831746    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-env-overrides\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:18.832240 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.831798    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-host-run-netns\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:18.832240 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.831829    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-multus-daemon-config\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:18.832240 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.831835    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-multus-cni-dir\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:18.832240 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.831827    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8187068e-17ec-4e6d-bb8e-1d0312aa6674-agent-certs\") pod \"konnectivity-agent-7p8pl\" (UID: \"8187068e-17ec-4e6d-bb8e-1d0312aa6674\") " pod="kube-system/konnectivity-agent-7p8pl"
Nov 03 22:12:18.832240 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.831884    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8187068e-17ec-4e6d-bb8e-1d0312aa6674-konnectivity-ca\") pod \"konnectivity-agent-7p8pl\" (UID: \"8187068e-17ec-4e6d-bb8e-1d0312aa6674\") " pod="kube-system/konnectivity-agent-7p8pl"
Nov 03 22:12:18.832240 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.831906    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-ovnkube-script-lib\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:18.832240 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.831910    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a06ec3e9-0a6e-46d8-ade5-13d657a6284c-run\") pod \"tuned-9ltv9\" (UID: \"a06ec3e9-0a6e-46d8-ade5-13d657a6284c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ltv9"
Nov 03 22:12:18.832240 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.831944    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fa1fd639-c265-4048-b9be-a583885ccb4e-socket-dir\") pod \"aws-ebs-csi-driver-node-8zcr2\" (UID: \"fa1fd639-c265-4048-b9be-a583885ccb4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zcr2"
Nov 03 22:12:18.832240 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.831986    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7l82\" (UniqueName: \"kubernetes.io/projected/a06ec3e9-0a6e-46d8-ade5-13d657a6284c-kube-api-access-t7l82\") pod \"tuned-9ltv9\" (UID: \"a06ec3e9-0a6e-46d8-ade5-13d657a6284c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ltv9"
Nov 03 22:12:18.832240 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.831988    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/089f9a4b-8eec-4758-811b-e506b41cecbb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4sm9h\" (UID: \"089f9a4b-8eec-4758-811b-e506b41cecbb\") " pod="openshift-multus/multus-additional-cni-plugins-4sm9h"
Nov 03 22:12:18.832240 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.831996    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-systemd-units\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:18.832240 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.832055    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-host-run-netns\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:18.832240 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.832124    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6x5m\" (UniqueName: \"kubernetes.io/projected/867a2a76-8aa3-4b07-98bf-913c2d2a5562-kube-api-access-f6x5m\") pod \"iptables-alerter-7rqgq\" (UID: \"867a2a76-8aa3-4b07-98bf-913c2d2a5562\") " pod="openshift-network-operator/iptables-alerter-7rqgq"
Nov 03 22:12:18.832240 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.832132    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a06ec3e9-0a6e-46d8-ade5-13d657a6284c-run\") pod \"tuned-9ltv9\" (UID: \"a06ec3e9-0a6e-46d8-ade5-13d657a6284c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ltv9"
Nov 03 22:12:18.832240 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.832168    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-hostroot\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:18.832240 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.832196    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/089f9a4b-8eec-4758-811b-e506b41cecbb-cni-binary-copy\") pod \"multus-additional-cni-plugins-4sm9h\" (UID: \"089f9a4b-8eec-4758-811b-e506b41cecbb\") " pod="openshift-multus/multus-additional-cni-plugins-4sm9h"
Nov 03 22:12:18.833086 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.832278    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bjhs\" (UniqueName: \"kubernetes.io/projected/089f9a4b-8eec-4758-811b-e506b41cecbb-kube-api-access-7bjhs\") pod \"multus-additional-cni-plugins-4sm9h\" (UID: \"089f9a4b-8eec-4758-811b-e506b41cecbb\") " pod="openshift-multus/multus-additional-cni-plugins-4sm9h"
Nov 03 22:12:18.833086 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.832306    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-node-log\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:18.833086 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.832339    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a06ec3e9-0a6e-46d8-ade5-13d657a6284c-etc-tuned\") pod \"tuned-9ltv9\" (UID: \"a06ec3e9-0a6e-46d8-ade5-13d657a6284c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ltv9"
Nov 03 22:12:18.833086 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.832356    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cc66a3de-7e7e-4855-8f17-db97dcfa815a-serviceca\") pod \"node-ca-p4p25\" (UID: \"cc66a3de-7e7e-4855-8f17-db97dcfa815a\") " pod="openshift-image-registry/node-ca-p4p25"
Nov 03 22:12:18.833086 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.832369    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-multus-socket-dir-parent\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:18.833086 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.832425    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-png47\" (UniqueName: \"kubernetes.io/projected/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-kube-api-access-png47\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:18.833086 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.832422    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-cni-binary-copy\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:18.833086 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.832463    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8ea0b360-1d72-4d40-935e-1394fecad74c-kubelet-config\") pod \"global-pull-secret-syncer-66m8v\" (UID: \"8ea0b360-1d72-4d40-935e-1394fecad74c\") " pod="kube-system/global-pull-secret-syncer-66m8v"
Nov 03 22:12:18.833086 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.832566    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a06ec3e9-0a6e-46d8-ade5-13d657a6284c-etc-kubernetes\") pod \"tuned-9ltv9\" (UID: \"a06ec3e9-0a6e-46d8-ade5-13d657a6284c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ltv9"
Nov 03 22:12:18.833086 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.832576    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-multus-socket-dir-parent\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:18.833086 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.832617    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8187068e-17ec-4e6d-bb8e-1d0312aa6674-konnectivity-ca\") pod \"konnectivity-agent-7p8pl\" (UID: \"8187068e-17ec-4e6d-bb8e-1d0312aa6674\") " pod="kube-system/konnectivity-agent-7p8pl"
Nov 03 22:12:18.833086 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.832670    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fa1fd639-c265-4048-b9be-a583885ccb4e-registration-dir\") pod \"aws-ebs-csi-driver-node-8zcr2\" (UID: \"fa1fd639-c265-4048-b9be-a583885ccb4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zcr2"
Nov 03 22:12:18.833086 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.832717    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-node-log\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:18.833086 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.832738    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a06ec3e9-0a6e-46d8-ade5-13d657a6284c-etc-kubernetes\") pod \"tuned-9ltv9\" (UID: \"a06ec3e9-0a6e-46d8-ade5-13d657a6284c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ltv9"
Nov 03 22:12:18.833086 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.832716    2570 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 22:12:18.833086 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.832802    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-host-kubelet\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:18.833086 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.832852    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-hostroot\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:18.833086 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.832904    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-var-lib-openvswitch\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:18.833926 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.832918    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8ea0b360-1d72-4d40-935e-1394fecad74c-kubelet-config\") pod \"global-pull-secret-syncer-66m8v\" (UID: \"8ea0b360-1d72-4d40-935e-1394fecad74c\") " pod="kube-system/global-pull-secret-syncer-66m8v"
Nov 03 22:12:18.833926 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.832926    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-var-lib-openvswitch\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:18.833926 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.832935    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-host-cni-bin\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:18.833926 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.832970    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-host-kubelet\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:18.833926 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.832984    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fa1fd639-c265-4048-b9be-a583885ccb4e-registration-dir\") pod \"aws-ebs-csi-driver-node-8zcr2\" (UID: \"fa1fd639-c265-4048-b9be-a583885ccb4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zcr2"
Nov 03 22:12:18.833926 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.833009    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-ovnkube-config\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:18.833926 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.833025    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-host-cni-bin\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:18.833926 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.833090    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-host-var-lib-kubelet\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:18.833926 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.833131    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94868020-2e98-4295-8db5-043461cc6e9d-tmp-dir\") pod \"node-resolver-x7cpm\" (UID: \"94868020-2e98-4295-8db5-043461cc6e9d\") " pod="openshift-dns/node-resolver-x7cpm"
Nov 03 22:12:18.833926 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.833160    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cc66a3de-7e7e-4855-8f17-db97dcfa815a-host\") pod \"node-ca-p4p25\" (UID: \"cc66a3de-7e7e-4855-8f17-db97dcfa815a\") " pod="openshift-image-registry/node-ca-p4p25"
Nov 03 22:12:18.833926 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.833178    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-host-var-lib-kubelet\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:18.833926 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.833219    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-etc-openvswitch\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:18.833926 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.833270    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a06ec3e9-0a6e-46d8-ade5-13d657a6284c-etc-sysctl-d\") pod \"tuned-9ltv9\" (UID: \"a06ec3e9-0a6e-46d8-ade5-13d657a6284c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ltv9"
Nov 03 22:12:18.833926 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.833307    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-host-run-k8s-cni-cncf-io\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:18.833926 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.833370    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8ea0b360-1d72-4d40-935e-1394fecad74c-dbus\") pod \"global-pull-secret-syncer-66m8v\" (UID: \"8ea0b360-1d72-4d40-935e-1394fecad74c\") " pod="kube-system/global-pull-secret-syncer-66m8v"
Nov 03 22:12:18.833926 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.833396    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ssqq\" (UniqueName: \"kubernetes.io/projected/cc66a3de-7e7e-4855-8f17-db97dcfa815a-kube-api-access-2ssqq\") pod \"node-ca-p4p25\" (UID: \"cc66a3de-7e7e-4855-8f17-db97dcfa815a\") " pod="openshift-image-registry/node-ca-p4p25"
Nov 03 22:12:18.833926 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.833412    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cc66a3de-7e7e-4855-8f17-db97dcfa815a-host\") pod \"node-ca-p4p25\" (UID: \"cc66a3de-7e7e-4855-8f17-db97dcfa815a\") " pod="openshift-image-registry/node-ca-p4p25"
Nov 03 22:12:18.833926 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.833419    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-host-slash\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:18.834750 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.833460    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/94868020-2e98-4295-8db5-043461cc6e9d-hosts-file\") pod \"node-resolver-x7cpm\" (UID: \"94868020-2e98-4295-8db5-043461cc6e9d\") " pod="openshift-dns/node-resolver-x7cpm"
Nov 03 22:12:18.834750 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.833451    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-host-slash\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:18.834750 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.833507    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fa1fd639-c265-4048-b9be-a583885ccb4e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8zcr2\" (UID: \"fa1fd639-c265-4048-b9be-a583885ccb4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zcr2"
Nov 03 22:12:18.834750 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.833519    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-host-run-k8s-cni-cncf-io\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:18.834750 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.833536    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fa1fd639-c265-4048-b9be-a583885ccb4e-sys-fs\") pod \"aws-ebs-csi-driver-node-8zcr2\" (UID: \"fa1fd639-c265-4048-b9be-a583885ccb4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zcr2"
Nov 03 22:12:18.834750 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.833565    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/089f9a4b-8eec-4758-811b-e506b41cecbb-system-cni-dir\") pod \"multus-additional-cni-plugins-4sm9h\" (UID: \"089f9a4b-8eec-4758-811b-e506b41cecbb\") " pod="openshift-multus/multus-additional-cni-plugins-4sm9h"
Nov 03 22:12:18.834750 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.833595    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/089f9a4b-8eec-4758-811b-e506b41cecbb-cnibin\") pod \"multus-additional-cni-plugins-4sm9h\" (UID: \"089f9a4b-8eec-4758-811b-e506b41cecbb\") " pod="openshift-multus/multus-additional-cni-plugins-4sm9h"
Nov 03 22:12:18.834750 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.833656    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/089f9a4b-8eec-4758-811b-e506b41cecbb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4sm9h\" (UID: \"089f9a4b-8eec-4758-811b-e506b41cecbb\") " pod="openshift-multus/multus-additional-cni-plugins-4sm9h"
Nov 03 22:12:18.834750 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.833689    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-host-cni-netd\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:18.834750 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.833715    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a06ec3e9-0a6e-46d8-ade5-13d657a6284c-lib-modules\") pod \"tuned-9ltv9\" (UID: \"a06ec3e9-0a6e-46d8-ade5-13d657a6284c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ltv9"
Nov 03 22:12:18.834750 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.833742    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/867a2a76-8aa3-4b07-98bf-913c2d2a5562-iptables-alerter-script\") pod \"iptables-alerter-7rqgq\" (UID: \"867a2a76-8aa3-4b07-98bf-913c2d2a5562\") " pod="openshift-network-operator/iptables-alerter-7rqgq"
Nov 03 22:12:18.834750 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.833769    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b071df2-94c8-4459-89f2-f4b448ceb581-metrics-certs\") pod \"network-metrics-daemon-jn5wf\" (UID: \"9b071df2-94c8-4459-89f2-f4b448ceb581\") " pod="openshift-multus/network-metrics-daemon-jn5wf"
Nov 03 22:12:18.834750 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.833784    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-ovnkube-config\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:18.834750 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.833799    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:18.834750 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.833786    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94868020-2e98-4295-8db5-043461cc6e9d-tmp-dir\") pod \"node-resolver-x7cpm\" (UID: \"94868020-2e98-4295-8db5-043461cc6e9d\") " pod="openshift-dns/node-resolver-x7cpm"
Nov 03 22:12:18.834750 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.833370    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/089f9a4b-8eec-4758-811b-e506b41cecbb-cni-binary-copy\") pod \"multus-additional-cni-plugins-4sm9h\" (UID: \"089f9a4b-8eec-4758-811b-e506b41cecbb\") " pod="openshift-multus/multus-additional-cni-plugins-4sm9h"
Nov 03 22:12:18.834750 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.833828    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrr2v\" (UniqueName: \"kubernetes.io/projected/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-kube-api-access-nrr2v\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:18.835610 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.833819    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-etc-openvswitch\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:18.835610 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.833906    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fa1fd639-c265-4048-b9be-a583885ccb4e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8zcr2\" (UID: \"fa1fd639-c265-4048-b9be-a583885ccb4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zcr2"
Nov 03 22:12:18.835610 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.833893    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:18.835610 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.833908    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fa1fd639-c265-4048-b9be-a583885ccb4e-sys-fs\") pod \"aws-ebs-csi-driver-node-8zcr2\" (UID: \"fa1fd639-c265-4048-b9be-a583885ccb4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zcr2"
Nov 03 22:12:18.835610 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.833934    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8ea0b360-1d72-4d40-935e-1394fecad74c-dbus\") pod \"global-pull-secret-syncer-66m8v\" (UID: \"8ea0b360-1d72-4d40-935e-1394fecad74c\") " pod="kube-system/global-pull-secret-syncer-66m8v"
Nov 03 22:12:18.835610 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.833937    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/089f9a4b-8eec-4758-811b-e506b41cecbb-system-cni-dir\") pod \"multus-additional-cni-plugins-4sm9h\" (UID: \"089f9a4b-8eec-4758-811b-e506b41cecbb\") " pod="openshift-multus/multus-additional-cni-plugins-4sm9h"
Nov 03 22:12:18.835610 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.834011    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a06ec3e9-0a6e-46d8-ade5-13d657a6284c-etc-sysctl-d\") pod \"tuned-9ltv9\" (UID: \"a06ec3e9-0a6e-46d8-ade5-13d657a6284c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ltv9"
Nov 03 22:12:18.835610 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.834036    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a06ec3e9-0a6e-46d8-ade5-13d657a6284c-lib-modules\") pod \"tuned-9ltv9\" (UID: \"a06ec3e9-0a6e-46d8-ade5-13d657a6284c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ltv9"
Nov 03 22:12:18.835610 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:18.834040    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 22:12:18.835610 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.834052    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/089f9a4b-8eec-4758-811b-e506b41cecbb-cnibin\") pod \"multus-additional-cni-plugins-4sm9h\" (UID: \"089f9a4b-8eec-4758-811b-e506b41cecbb\") " pod="openshift-multus/multus-additional-cni-plugins-4sm9h"
Nov 03 22:12:18.835610 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:18.834095    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b071df2-94c8-4459-89f2-f4b448ceb581-metrics-certs podName:9b071df2-94c8-4459-89f2-f4b448ceb581 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:19.334077672 +0000 UTC m=+2.045141103 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9b071df2-94c8-4459-89f2-f4b448ceb581-metrics-certs") pod "network-metrics-daemon-jn5wf" (UID: "9b071df2-94c8-4459-89f2-f4b448ceb581") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 22:12:18.835610 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.834097    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/94868020-2e98-4295-8db5-043461cc6e9d-hosts-file\") pod \"node-resolver-x7cpm\" (UID: \"94868020-2e98-4295-8db5-043461cc6e9d\") " pod="openshift-dns/node-resolver-x7cpm"
Nov 03 22:12:18.835610 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.834135    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a06ec3e9-0a6e-46d8-ade5-13d657a6284c-etc-systemd\") pod \"tuned-9ltv9\" (UID: \"a06ec3e9-0a6e-46d8-ade5-13d657a6284c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ltv9"
Nov 03 22:12:18.835610 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.834187    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a06ec3e9-0a6e-46d8-ade5-13d657a6284c-etc-systemd\") pod \"tuned-9ltv9\" (UID: \"a06ec3e9-0a6e-46d8-ade5-13d657a6284c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ltv9"
Nov 03 22:12:18.835610 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.834187    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-host-cni-netd\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:18.835610 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.834202    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a06ec3e9-0a6e-46d8-ade5-13d657a6284c-sys\") pod \"tuned-9ltv9\" (UID: \"a06ec3e9-0a6e-46d8-ade5-13d657a6284c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ltv9"
Nov 03 22:12:18.835610 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.834222    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-host-run-multus-certs\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:18.836443 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.834256    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shrkm\" (UniqueName: \"kubernetes.io/projected/fa1fd639-c265-4048-b9be-a583885ccb4e-kube-api-access-shrkm\") pod \"aws-ebs-csi-driver-node-8zcr2\" (UID: \"fa1fd639-c265-4048-b9be-a583885ccb4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zcr2"
Nov 03 22:12:18.836443 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.834279    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/089f9a4b-8eec-4758-811b-e506b41cecbb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4sm9h\" (UID: \"089f9a4b-8eec-4758-811b-e506b41cecbb\") " pod="openshift-multus/multus-additional-cni-plugins-4sm9h"
Nov 03 22:12:18.836443 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.834319    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-run-systemd\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:18.836443 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.834313    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-host-run-multus-certs\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:18.836443 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.835073    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a06ec3e9-0a6e-46d8-ade5-13d657a6284c-sys\") pod \"tuned-9ltv9\" (UID: \"a06ec3e9-0a6e-46d8-ade5-13d657a6284c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ltv9"
Nov 03 22:12:18.836443 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.835140    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/867a2a76-8aa3-4b07-98bf-913c2d2a5562-host-slash\") pod \"iptables-alerter-7rqgq\" (UID: \"867a2a76-8aa3-4b07-98bf-913c2d2a5562\") " pod="openshift-network-operator/iptables-alerter-7rqgq"
Nov 03 22:12:18.836443 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.835217    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-host-var-lib-cni-multus\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:18.836443 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.835256    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-multus-conf-dir\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:18.836443 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.835331    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8ea0b360-1d72-4d40-935e-1394fecad74c-original-pull-secret\") pod \"global-pull-secret-syncer-66m8v\" (UID: \"8ea0b360-1d72-4d40-935e-1394fecad74c\") " pod="kube-system/global-pull-secret-syncer-66m8v"
Nov 03 22:12:18.836443 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.835374    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/867a2a76-8aa3-4b07-98bf-913c2d2a5562-iptables-alerter-script\") pod \"iptables-alerter-7rqgq\" (UID: \"867a2a76-8aa3-4b07-98bf-913c2d2a5562\") " pod="openshift-network-operator/iptables-alerter-7rqgq"
Nov 03 22:12:18.836443 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.835423    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-run-ovn\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:18.836443 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.835457    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-log-socket\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:18.836443 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.835459    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/867a2a76-8aa3-4b07-98bf-913c2d2a5562-host-slash\") pod \"iptables-alerter-7rqgq\" (UID: \"867a2a76-8aa3-4b07-98bf-913c2d2a5562\") " pod="openshift-network-operator/iptables-alerter-7rqgq"
Nov 03 22:12:18.836443 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.835517    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-host-var-lib-cni-multus\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:18.836443 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:18.835533    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 22:12:18.836443 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.835607    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-multus-conf-dir\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:18.836443 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:18.835617    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ea0b360-1d72-4d40-935e-1394fecad74c-original-pull-secret podName:8ea0b360-1d72-4d40-935e-1394fecad74c nodeName:}" failed. No retries permitted until 2026-11-03 22:12:19.335597098 +0000 UTC m=+2.046660518 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8ea0b360-1d72-4d40-935e-1394fecad74c-original-pull-secret") pod "global-pull-secret-syncer-66m8v" (UID: "8ea0b360-1d72-4d40-935e-1394fecad74c") : object "kube-system"/"original-pull-secret" not registered
Nov 03 22:12:18.837215 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.835676    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a06ec3e9-0a6e-46d8-ade5-13d657a6284c-host\") pod \"tuned-9ltv9\" (UID: \"a06ec3e9-0a6e-46d8-ade5-13d657a6284c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ltv9"
Nov 03 22:12:18.837215 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.835691    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-run-ovn\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:18.837215 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.835730    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a06ec3e9-0a6e-46d8-ade5-13d657a6284c-etc-sysconfig\") pod \"tuned-9ltv9\" (UID: \"a06ec3e9-0a6e-46d8-ade5-13d657a6284c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ltv9"
Nov 03 22:12:18.837215 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.835739    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-log-socket\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:18.837215 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.835774    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-run-systemd\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:18.837215 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.835811    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a06ec3e9-0a6e-46d8-ade5-13d657a6284c-tmp\") pod \"tuned-9ltv9\" (UID: \"a06ec3e9-0a6e-46d8-ade5-13d657a6284c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ltv9"
Nov 03 22:12:18.837215 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.835851    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-system-cni-dir\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:18.837215 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.835857    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a06ec3e9-0a6e-46d8-ade5-13d657a6284c-etc-sysconfig\") pod \"tuned-9ltv9\" (UID: \"a06ec3e9-0a6e-46d8-ade5-13d657a6284c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ltv9"
Nov 03 22:12:18.837215 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.835885    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fa1fd639-c265-4048-b9be-a583885ccb4e-device-dir\") pod \"aws-ebs-csi-driver-node-8zcr2\" (UID: \"fa1fd639-c265-4048-b9be-a583885ccb4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zcr2"
Nov 03 22:12:18.837215 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.836288    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-host-run-netns\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:18.837215 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.836370    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a06ec3e9-0a6e-46d8-ade5-13d657a6284c-var-lib-kubelet\") pod \"tuned-9ltv9\" (UID: \"a06ec3e9-0a6e-46d8-ade5-13d657a6284c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ltv9"
Nov 03 22:12:18.837215 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.836373    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a06ec3e9-0a6e-46d8-ade5-13d657a6284c-host\") pod \"tuned-9ltv9\" (UID: \"a06ec3e9-0a6e-46d8-ade5-13d657a6284c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ltv9"
Nov 03 22:12:18.837215 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.836417    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-cnibin\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:18.837215 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.836441    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a06ec3e9-0a6e-46d8-ade5-13d657a6284c-var-lib-kubelet\") pod \"tuned-9ltv9\" (UID: \"a06ec3e9-0a6e-46d8-ade5-13d657a6284c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ltv9"
Nov 03 22:12:18.837215 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.836708    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fa1fd639-c265-4048-b9be-a583885ccb4e-etc-selinux\") pod \"aws-ebs-csi-driver-node-8zcr2\" (UID: \"fa1fd639-c265-4048-b9be-a583885ccb4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zcr2"
Nov 03 22:12:18.837215 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.837103    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-cnibin\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:18.837215 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.837169    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fa1fd639-c265-4048-b9be-a583885ccb4e-device-dir\") pod \"aws-ebs-csi-driver-node-8zcr2\" (UID: \"fa1fd639-c265-4048-b9be-a583885ccb4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zcr2"
Nov 03 22:12:18.838142 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.837507    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-ovn-node-metrics-cert\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:18.838142 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.837918    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-system-cni-dir\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:18.838142 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.838021    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8187068e-17ec-4e6d-bb8e-1d0312aa6674-agent-certs\") pod \"konnectivity-agent-7p8pl\" (UID: \"8187068e-17ec-4e6d-bb8e-1d0312aa6674\") " pod="kube-system/konnectivity-agent-7p8pl"
Nov 03 22:12:18.838142 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.838137    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a06ec3e9-0a6e-46d8-ade5-13d657a6284c-etc-tuned\") pod \"tuned-9ltv9\" (UID: \"a06ec3e9-0a6e-46d8-ade5-13d657a6284c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ltv9"
Nov 03 22:12:18.839181 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.838642    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/089f9a4b-8eec-4758-811b-e506b41cecbb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4sm9h\" (UID: \"089f9a4b-8eec-4758-811b-e506b41cecbb\") " pod="openshift-multus/multus-additional-cni-plugins-4sm9h"
Nov 03 22:12:18.839181 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.838777    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/089f9a4b-8eec-4758-811b-e506b41cecbb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4sm9h\" (UID: \"089f9a4b-8eec-4758-811b-e506b41cecbb\") " pod="openshift-multus/multus-additional-cni-plugins-4sm9h"
Nov 03 22:12:18.840214 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.840192    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-host-run-netns\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:18.840214 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.840196    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a06ec3e9-0a6e-46d8-ade5-13d657a6284c-tmp\") pod \"tuned-9ltv9\" (UID: \"a06ec3e9-0a6e-46d8-ade5-13d657a6284c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ltv9"
Nov 03 22:12:18.840400 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.840262    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fa1fd639-c265-4048-b9be-a583885ccb4e-etc-selinux\") pod \"aws-ebs-csi-driver-node-8zcr2\" (UID: \"fa1fd639-c265-4048-b9be-a583885ccb4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zcr2"
Nov 03 22:12:18.852470 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:18.852442    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 22:12:18.852470 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:18.852464    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 22:12:18.852631 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:18.852476    2570 projected.go:196] Error preparing data for projected volume kube-api-access-qdtpv for pod openshift-network-diagnostics/network-check-target-82h4r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 22:12:18.852631 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:18.852527    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b792dc0-5e1c-40c4-a6c6-f68b4c6c8aac-kube-api-access-qdtpv podName:1b792dc0-5e1c-40c4-a6c6-f68b4c6c8aac nodeName:}" failed. No retries permitted until 2026-11-03 22:12:19.352514691 +0000 UTC m=+2.063578131 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-qdtpv" (UniqueName: "kubernetes.io/projected/1b792dc0-5e1c-40c4-a6c6-f68b4c6c8aac-kube-api-access-qdtpv") pod "network-check-target-82h4r" (UID: "1b792dc0-5e1c-40c4-a6c6-f68b4c6c8aac") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 22:12:18.854334 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.854312    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhr4l\" (UniqueName: \"kubernetes.io/projected/9b071df2-94c8-4459-89f2-f4b448ceb581-kube-api-access-mhr4l\") pod \"network-metrics-daemon-jn5wf\" (UID: \"9b071df2-94c8-4459-89f2-f4b448ceb581\") " pod="openshift-multus/network-metrics-daemon-jn5wf"
Nov 03 22:12:18.855203 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.855183    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bjhs\" (UniqueName: \"kubernetes.io/projected/089f9a4b-8eec-4758-811b-e506b41cecbb-kube-api-access-7bjhs\") pod \"multus-additional-cni-plugins-4sm9h\" (UID: \"089f9a4b-8eec-4758-811b-e506b41cecbb\") " pod="openshift-multus/multus-additional-cni-plugins-4sm9h"
Nov 03 22:12:18.855432 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.855414    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-png47\" (UniqueName: \"kubernetes.io/projected/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-kube-api-access-png47\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:18.855936 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.855904    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fswpg\" (UniqueName: \"kubernetes.io/projected/94868020-2e98-4295-8db5-043461cc6e9d-kube-api-access-fswpg\") pod \"node-resolver-x7cpm\" (UID: \"94868020-2e98-4295-8db5-043461cc6e9d\") " pod="openshift-dns/node-resolver-x7cpm"
Nov 03 22:12:18.856983 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.856961    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7l82\" (UniqueName: \"kubernetes.io/projected/a06ec3e9-0a6e-46d8-ade5-13d657a6284c-kube-api-access-t7l82\") pod \"tuned-9ltv9\" (UID: \"a06ec3e9-0a6e-46d8-ade5-13d657a6284c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ltv9"
Nov 03 22:12:18.859416 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.859396    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6x5m\" (UniqueName: \"kubernetes.io/projected/867a2a76-8aa3-4b07-98bf-913c2d2a5562-kube-api-access-f6x5m\") pod \"iptables-alerter-7rqgq\" (UID: \"867a2a76-8aa3-4b07-98bf-913c2d2a5562\") " pod="openshift-network-operator/iptables-alerter-7rqgq"
Nov 03 22:12:18.863205 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.863186    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ssqq\" (UniqueName: \"kubernetes.io/projected/cc66a3de-7e7e-4855-8f17-db97dcfa815a-kube-api-access-2ssqq\") pod \"node-ca-p4p25\" (UID: \"cc66a3de-7e7e-4855-8f17-db97dcfa815a\") " pod="openshift-image-registry/node-ca-p4p25"
Nov 03 22:12:18.863356 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.863336    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shrkm\" (UniqueName: \"kubernetes.io/projected/fa1fd639-c265-4048-b9be-a583885ccb4e-kube-api-access-shrkm\") pod \"aws-ebs-csi-driver-node-8zcr2\" (UID: \"fa1fd639-c265-4048-b9be-a583885ccb4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zcr2"
Nov 03 22:12:18.863513 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.863497    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrr2v\" (UniqueName: \"kubernetes.io/projected/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-kube-api-access-nrr2v\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:18.872301 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.872273    2570 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 22:12:18.983329 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.983305    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-x7cpm"
Nov 03 22:12:18.993952 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:18.993928    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94868020_2e98_4295_8db5_043461cc6e9d.slice/crio-69a840eb07f498030ab8307ebdd6e0e3f3dd7188385aa3f7dec3998e7b8b3c21 WatchSource:0}: Error finding container 69a840eb07f498030ab8307ebdd6e0e3f3dd7188385aa3f7dec3998e7b8b3c21: Status 404 returned error can't find the container with id 69a840eb07f498030ab8307ebdd6e0e3f3dd7188385aa3f7dec3998e7b8b3c21
Nov 03 22:12:18.999454 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:18.999434    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-p4p25"
Nov 03 22:12:19.026281 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:19.026245    2570 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 22:12:19.032686 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:19.032669    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7p8pl"
Nov 03 22:12:19.033837 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:19.033818    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9ltv9"
Nov 03 22:12:19.040418 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:19.040400    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:19.042724 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:19.042701    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8187068e_17ec_4e6d_bb8e_1d0312aa6674.slice/crio-46377aeffc4cd3d572a6c7b722747c8c24bbebb5583a67d3a04edfb6df236e07 WatchSource:0}: Error finding container 46377aeffc4cd3d572a6c7b722747c8c24bbebb5583a67d3a04edfb6df236e07: Status 404 returned error can't find the container with id 46377aeffc4cd3d572a6c7b722747c8c24bbebb5583a67d3a04edfb6df236e07
Nov 03 22:12:19.044690 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:19.044670    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda06ec3e9_0a6e_46d8_ade5_13d657a6284c.slice/crio-72300af8e137757d0f41a0b8c50a35d133306acc9f5024f99885890a9b314d92 WatchSource:0}: Error finding container 72300af8e137757d0f41a0b8c50a35d133306acc9f5024f99885890a9b314d92: Status 404 returned error can't find the container with id 72300af8e137757d0f41a0b8c50a35d133306acc9f5024f99885890a9b314d92
Nov 03 22:12:19.047881 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:19.047864    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zcr2"
Nov 03 22:12:19.050875 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:19.050856    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7rqgq"
Nov 03 22:12:19.051310 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:19.051286    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba86cda4_1b8a_413e_b7ae_0fa9b281d0ea.slice/crio-382c0ab10609a37b4a1f8e32d5dac3760c552c63c3a60682227fb3bfdaafca3c WatchSource:0}: Error finding container 382c0ab10609a37b4a1f8e32d5dac3760c552c63c3a60682227fb3bfdaafca3c: Status 404 returned error can't find the container with id 382c0ab10609a37b4a1f8e32d5dac3760c552c63c3a60682227fb3bfdaafca3c
Nov 03 22:12:19.057955 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:19.057933    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa1fd639_c265_4048_b9be_a583885ccb4e.slice/crio-5a9c9ef3d251d426e883ac5835699c7f59588d1b12d6f68ced725e4607dbdaf5 WatchSource:0}: Error finding container 5a9c9ef3d251d426e883ac5835699c7f59588d1b12d6f68ced725e4607dbdaf5: Status 404 returned error can't find the container with id 5a9c9ef3d251d426e883ac5835699c7f59588d1b12d6f68ced725e4607dbdaf5
Nov 03 22:12:19.060487 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:19.060419    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:19.062190 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:19.062163    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod867a2a76_8aa3_4b07_98bf_913c2d2a5562.slice/crio-7b9044ebe11612a0e9eded07aab7df7c563a18a145b86286b40ed006dfb27949 WatchSource:0}: Error finding container 7b9044ebe11612a0e9eded07aab7df7c563a18a145b86286b40ed006dfb27949: Status 404 returned error can't find the container with id 7b9044ebe11612a0e9eded07aab7df7c563a18a145b86286b40ed006dfb27949
Nov 03 22:12:19.069991 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:19.069970    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac5ecdf9_3675_4d4f_98e3_a4e8796669be.slice/crio-9427fc09e99178a5f76492a05fc10f074027d4022ad640cc3a535c624220f6d8 WatchSource:0}: Error finding container 9427fc09e99178a5f76492a05fc10f074027d4022ad640cc3a535c624220f6d8: Status 404 returned error can't find the container with id 9427fc09e99178a5f76492a05fc10f074027d4022ad640cc3a535c624220f6d8
Nov 03 22:12:19.075163 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:19.075143    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4sm9h"
Nov 03 22:12:19.084845 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:19.084824    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod089f9a4b_8eec_4758_811b_e506b41cecbb.slice/crio-01527eb9e50404bb5f91b695b61e185b6e7cf0927fd0d1252a2bb9d0c36ddc0e WatchSource:0}: Error finding container 01527eb9e50404bb5f91b695b61e185b6e7cf0927fd0d1252a2bb9d0c36ddc0e: Status 404 returned error can't find the container with id 01527eb9e50404bb5f91b695b61e185b6e7cf0927fd0d1252a2bb9d0c36ddc0e
Nov 03 22:12:19.345635 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:19.345557    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8ea0b360-1d72-4d40-935e-1394fecad74c-original-pull-secret\") pod \"global-pull-secret-syncer-66m8v\" (UID: \"8ea0b360-1d72-4d40-935e-1394fecad74c\") " pod="kube-system/global-pull-secret-syncer-66m8v"
Nov 03 22:12:19.345791 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:19.345640    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b071df2-94c8-4459-89f2-f4b448ceb581-metrics-certs\") pod \"network-metrics-daemon-jn5wf\" (UID: \"9b071df2-94c8-4459-89f2-f4b448ceb581\") " pod="openshift-multus/network-metrics-daemon-jn5wf"
Nov 03 22:12:19.345791 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:19.345767    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 22:12:19.345867 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:19.345824    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b071df2-94c8-4459-89f2-f4b448ceb581-metrics-certs podName:9b071df2-94c8-4459-89f2-f4b448ceb581 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:20.345810327 +0000 UTC m=+3.056873756 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9b071df2-94c8-4459-89f2-f4b448ceb581-metrics-certs") pod "network-metrics-daemon-jn5wf" (UID: "9b071df2-94c8-4459-89f2-f4b448ceb581") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 22:12:19.346519 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:19.346484    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 22:12:19.346635 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:19.346605    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ea0b360-1d72-4d40-935e-1394fecad74c-original-pull-secret podName:8ea0b360-1d72-4d40-935e-1394fecad74c nodeName:}" failed. No retries permitted until 2026-11-03 22:12:20.346576429 +0000 UTC m=+3.057639861 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8ea0b360-1d72-4d40-935e-1394fecad74c-original-pull-secret") pod "global-pull-secret-syncer-66m8v" (UID: "8ea0b360-1d72-4d40-935e-1394fecad74c") : object "kube-system"/"original-pull-secret" not registered
Nov 03 22:12:19.446702 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:19.446654    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdtpv\" (UniqueName: \"kubernetes.io/projected/1b792dc0-5e1c-40c4-a6c6-f68b4c6c8aac-kube-api-access-qdtpv\") pod \"network-check-target-82h4r\" (UID: \"1b792dc0-5e1c-40c4-a6c6-f68b4c6c8aac\") " pod="openshift-network-diagnostics/network-check-target-82h4r"
Nov 03 22:12:19.446885 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:19.446845    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 22:12:19.446885 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:19.446873    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 22:12:19.446885 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:19.446887    2570 projected.go:196] Error preparing data for projected volume kube-api-access-qdtpv for pod openshift-network-diagnostics/network-check-target-82h4r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 22:12:19.447052 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:19.446948    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b792dc0-5e1c-40c4-a6c6-f68b4c6c8aac-kube-api-access-qdtpv podName:1b792dc0-5e1c-40c4-a6c6-f68b4c6c8aac nodeName:}" failed. No retries permitted until 2026-11-03 22:12:20.446929904 +0000 UTC m=+3.157993325 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-qdtpv" (UniqueName: "kubernetes.io/projected/1b792dc0-5e1c-40c4-a6c6-f68b4c6c8aac-kube-api-access-qdtpv") pod "network-check-target-82h4r" (UID: "1b792dc0-5e1c-40c4-a6c6-f68b4c6c8aac") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 22:12:19.712981 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:19.712835    2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 22:07:18 +0000 UTC" deadline="2028-06-09 12:12:28.219525608 +0000 UTC"
Nov 03 22:12:19.712981 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:19.712931    2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14006h0m8.506599046s"
Nov 03 22:12:19.751969 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:19.751907    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-66m8v"
Nov 03 22:12:19.752145 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:19.752051    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-66m8v" podUID="8ea0b360-1d72-4d40-935e-1394fecad74c"
Nov 03 22:12:19.786496 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:19.786434    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4sm9h" event={"ID":"089f9a4b-8eec-4758-811b-e506b41cecbb","Type":"ContainerStarted","Data":"01527eb9e50404bb5f91b695b61e185b6e7cf0927fd0d1252a2bb9d0c36ddc0e"}
Nov 03 22:12:19.794333 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:19.794273    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7rqgq" event={"ID":"867a2a76-8aa3-4b07-98bf-913c2d2a5562","Type":"ContainerStarted","Data":"7b9044ebe11612a0e9eded07aab7df7c563a18a145b86286b40ed006dfb27949"}
Nov 03 22:12:19.800887 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:19.800852    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8" event={"ID":"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea","Type":"ContainerStarted","Data":"382c0ab10609a37b4a1f8e32d5dac3760c552c63c3a60682227fb3bfdaafca3c"}
Nov 03 22:12:19.811723 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:19.811687    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9ltv9" event={"ID":"a06ec3e9-0a6e-46d8-ade5-13d657a6284c","Type":"ContainerStarted","Data":"72300af8e137757d0f41a0b8c50a35d133306acc9f5024f99885890a9b314d92"}
Nov 03 22:12:19.822383 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:19.822328    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7p8pl" event={"ID":"8187068e-17ec-4e6d-bb8e-1d0312aa6674","Type":"ContainerStarted","Data":"46377aeffc4cd3d572a6c7b722747c8c24bbebb5583a67d3a04edfb6df236e07"}
Nov 03 22:12:19.846996 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:19.846955    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-p4p25" event={"ID":"cc66a3de-7e7e-4855-8f17-db97dcfa815a","Type":"ContainerStarted","Data":"d02c882efc6b58702c439be61f776a0b522e52d436f895cce65cb3ff7d585d6b"}
Nov 03 22:12:19.894085 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:19.894040    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-x7cpm" event={"ID":"94868020-2e98-4295-8db5-043461cc6e9d","Type":"ContainerStarted","Data":"69a840eb07f498030ab8307ebdd6e0e3f3dd7188385aa3f7dec3998e7b8b3c21"}
Nov 03 22:12:19.900784 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:19.900748    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2k9fs" event={"ID":"ac5ecdf9-3675-4d4f-98e3-a4e8796669be","Type":"ContainerStarted","Data":"9427fc09e99178a5f76492a05fc10f074027d4022ad640cc3a535c624220f6d8"}
Nov 03 22:12:19.910084 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:19.910033    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zcr2" event={"ID":"fa1fd639-c265-4048-b9be-a583885ccb4e","Type":"ContainerStarted","Data":"5a9c9ef3d251d426e883ac5835699c7f59588d1b12d6f68ced725e4607dbdaf5"}
Nov 03 22:12:20.358068 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:20.358029    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b071df2-94c8-4459-89f2-f4b448ceb581-metrics-certs\") pod \"network-metrics-daemon-jn5wf\" (UID: \"9b071df2-94c8-4459-89f2-f4b448ceb581\") " pod="openshift-multus/network-metrics-daemon-jn5wf"
Nov 03 22:12:20.358278 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:20.358088    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8ea0b360-1d72-4d40-935e-1394fecad74c-original-pull-secret\") pod \"global-pull-secret-syncer-66m8v\" (UID: \"8ea0b360-1d72-4d40-935e-1394fecad74c\") " pod="kube-system/global-pull-secret-syncer-66m8v"
Nov 03 22:12:20.358278 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:20.358272    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 22:12:20.358407 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:20.358339    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ea0b360-1d72-4d40-935e-1394fecad74c-original-pull-secret podName:8ea0b360-1d72-4d40-935e-1394fecad74c nodeName:}" failed. No retries permitted until 2026-11-03 22:12:22.358320002 +0000 UTC m=+5.069383434 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8ea0b360-1d72-4d40-935e-1394fecad74c-original-pull-secret") pod "global-pull-secret-syncer-66m8v" (UID: "8ea0b360-1d72-4d40-935e-1394fecad74c") : object "kube-system"/"original-pull-secret" not registered
Nov 03 22:12:20.358756 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:20.358737    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 22:12:20.358846 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:20.358792    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b071df2-94c8-4459-89f2-f4b448ceb581-metrics-certs podName:9b071df2-94c8-4459-89f2-f4b448ceb581 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:22.35877536 +0000 UTC m=+5.069838790 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9b071df2-94c8-4459-89f2-f4b448ceb581-metrics-certs") pod "network-metrics-daemon-jn5wf" (UID: "9b071df2-94c8-4459-89f2-f4b448ceb581") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 22:12:20.458634 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:20.458596    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdtpv\" (UniqueName: \"kubernetes.io/projected/1b792dc0-5e1c-40c4-a6c6-f68b4c6c8aac-kube-api-access-qdtpv\") pod \"network-check-target-82h4r\" (UID: \"1b792dc0-5e1c-40c4-a6c6-f68b4c6c8aac\") " pod="openshift-network-diagnostics/network-check-target-82h4r"
Nov 03 22:12:20.458820 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:20.458777    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 22:12:20.458820 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:20.458796    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 22:12:20.458820 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:20.458809    2570 projected.go:196] Error preparing data for projected volume kube-api-access-qdtpv for pod openshift-network-diagnostics/network-check-target-82h4r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 22:12:20.458990 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:20.458866    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b792dc0-5e1c-40c4-a6c6-f68b4c6c8aac-kube-api-access-qdtpv podName:1b792dc0-5e1c-40c4-a6c6-f68b4c6c8aac nodeName:}" failed. No retries permitted until 2026-11-03 22:12:22.458848121 +0000 UTC m=+5.169911541 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-qdtpv" (UniqueName: "kubernetes.io/projected/1b792dc0-5e1c-40c4-a6c6-f68b4c6c8aac-kube-api-access-qdtpv") pod "network-check-target-82h4r" (UID: "1b792dc0-5e1c-40c4-a6c6-f68b4c6c8aac") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 22:12:20.715105 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:20.714415    2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 22:07:18 +0000 UTC" deadline="2028-07-16 07:43:08.748767808 +0000 UTC"
Nov 03 22:12:20.715105 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:20.714461    2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14889h30m48.034314675s"
Nov 03 22:12:20.750335 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:20.750299    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jn5wf"
Nov 03 22:12:20.750527 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:20.750464    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jn5wf" podUID="9b071df2-94c8-4459-89f2-f4b448ceb581"
Nov 03 22:12:20.750927 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:20.750901    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-82h4r"
Nov 03 22:12:20.751034 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:20.751011    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-82h4r" podUID="1b792dc0-5e1c-40c4-a6c6-f68b4c6c8aac"
Nov 03 22:12:21.753451 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:21.752927    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-66m8v"
Nov 03 22:12:21.753451 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:21.753067    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-66m8v" podUID="8ea0b360-1d72-4d40-935e-1394fecad74c"
Nov 03 22:12:22.375120 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:22.375080    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b071df2-94c8-4459-89f2-f4b448ceb581-metrics-certs\") pod \"network-metrics-daemon-jn5wf\" (UID: \"9b071df2-94c8-4459-89f2-f4b448ceb581\") " pod="openshift-multus/network-metrics-daemon-jn5wf"
Nov 03 22:12:22.375314 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:22.375138    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8ea0b360-1d72-4d40-935e-1394fecad74c-original-pull-secret\") pod \"global-pull-secret-syncer-66m8v\" (UID: \"8ea0b360-1d72-4d40-935e-1394fecad74c\") " pod="kube-system/global-pull-secret-syncer-66m8v"
Nov 03 22:12:22.375314 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:22.375296    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 22:12:22.375425 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:22.375358    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ea0b360-1d72-4d40-935e-1394fecad74c-original-pull-secret podName:8ea0b360-1d72-4d40-935e-1394fecad74c nodeName:}" failed. No retries permitted until 2026-11-03 22:12:26.375341361 +0000 UTC m=+9.086404779 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8ea0b360-1d72-4d40-935e-1394fecad74c-original-pull-secret") pod "global-pull-secret-syncer-66m8v" (UID: "8ea0b360-1d72-4d40-935e-1394fecad74c") : object "kube-system"/"original-pull-secret" not registered
Nov 03 22:12:22.375788 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:22.375760    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 22:12:22.375900 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:22.375813    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b071df2-94c8-4459-89f2-f4b448ceb581-metrics-certs podName:9b071df2-94c8-4459-89f2-f4b448ceb581 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:26.375798798 +0000 UTC m=+9.086862212 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9b071df2-94c8-4459-89f2-f4b448ceb581-metrics-certs") pod "network-metrics-daemon-jn5wf" (UID: "9b071df2-94c8-4459-89f2-f4b448ceb581") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 22:12:22.476247 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:22.476198    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdtpv\" (UniqueName: \"kubernetes.io/projected/1b792dc0-5e1c-40c4-a6c6-f68b4c6c8aac-kube-api-access-qdtpv\") pod \"network-check-target-82h4r\" (UID: \"1b792dc0-5e1c-40c4-a6c6-f68b4c6c8aac\") " pod="openshift-network-diagnostics/network-check-target-82h4r"
Nov 03 22:12:22.476399 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:22.476373    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 22:12:22.476399 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:22.476392    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 22:12:22.476522 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:22.476404    2570 projected.go:196] Error preparing data for projected volume kube-api-access-qdtpv for pod openshift-network-diagnostics/network-check-target-82h4r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 22:12:22.476522 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:22.476459    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b792dc0-5e1c-40c4-a6c6-f68b4c6c8aac-kube-api-access-qdtpv podName:1b792dc0-5e1c-40c4-a6c6-f68b4c6c8aac nodeName:}" failed. No retries permitted until 2026-11-03 22:12:26.476441679 +0000 UTC m=+9.187505110 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-qdtpv" (UniqueName: "kubernetes.io/projected/1b792dc0-5e1c-40c4-a6c6-f68b4c6c8aac-kube-api-access-qdtpv") pod "network-check-target-82h4r" (UID: "1b792dc0-5e1c-40c4-a6c6-f68b4c6c8aac") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 22:12:22.750543 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:22.750098    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jn5wf"
Nov 03 22:12:22.750543 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:22.750264    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jn5wf" podUID="9b071df2-94c8-4459-89f2-f4b448ceb581"
Nov 03 22:12:22.750724 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:22.750634    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-82h4r"
Nov 03 22:12:22.750773 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:22.750723    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-82h4r" podUID="1b792dc0-5e1c-40c4-a6c6-f68b4c6c8aac"
Nov 03 22:12:22.945245 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:22.944936    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4sm9h" event={"ID":"089f9a4b-8eec-4758-811b-e506b41cecbb","Type":"ContainerStarted","Data":"1a0d137e44a33d2070392ea05e7146e8b051f19e723dacdaaf22b4d51b221b01"}
Nov 03 22:12:23.752803 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:23.752423    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-66m8v"
Nov 03 22:12:23.752803 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:23.752542    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-66m8v" podUID="8ea0b360-1d72-4d40-935e-1394fecad74c"
Nov 03 22:12:23.953563 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:23.953483    2570 generic.go:356] "Generic (PLEG): container finished" podID="089f9a4b-8eec-4758-811b-e506b41cecbb" containerID="1a0d137e44a33d2070392ea05e7146e8b051f19e723dacdaaf22b4d51b221b01" exitCode=0
Nov 03 22:12:23.954008 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:23.953575    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4sm9h" event={"ID":"089f9a4b-8eec-4758-811b-e506b41cecbb","Type":"ContainerDied","Data":"1a0d137e44a33d2070392ea05e7146e8b051f19e723dacdaaf22b4d51b221b01"}
Nov 03 22:12:23.959182 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:23.958714    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7p8pl" event={"ID":"8187068e-17ec-4e6d-bb8e-1d0312aa6674","Type":"ContainerStarted","Data":"722eb56c241d8217bfca9cdcb90207ed411113c7ecda58777eac9dda32b526e3"}
Nov 03 22:12:23.961513 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:23.961092    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-73.ec2.internal" event={"ID":"a7264ab43e0f926a44edf6ae184aafb2","Type":"ContainerStarted","Data":"5d6dd3ebaab3f6b7146fb4137385e43916a5fdd275c604e8d710477b733185de"}
Nov 03 22:12:23.964291 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:23.963725    2570 generic.go:356] "Generic (PLEG): container finished" podID="9e2325705fcd5fefcf1f90504938970b" containerID="4218d9aba019bb13e1453000949158430ba6f312c71019cd678a3464a8f93fbe" exitCode=0
Nov 03 22:12:23.964291 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:23.963762    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-73.ec2.internal" event={"ID":"9e2325705fcd5fefcf1f90504938970b","Type":"ContainerDied","Data":"4218d9aba019bb13e1453000949158430ba6f312c71019cd678a3464a8f93fbe"}
Nov 03 22:12:24.081331 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:24.080195    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-73.ec2.internal" podStartSLOduration=6.080178444 podStartE2EDuration="6.080178444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:12:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:24.041515858 +0000 UTC m=+6.752579295" watchObservedRunningTime="2026-11-03 22:12:24.080178444 +0000 UTC m=+6.791241881"
Nov 03 22:12:24.750185 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:24.750155    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jn5wf"
Nov 03 22:12:24.750349 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:24.750321    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jn5wf" podUID="9b071df2-94c8-4459-89f2-f4b448ceb581"
Nov 03 22:12:24.750425 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:24.750410    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-82h4r"
Nov 03 22:12:24.750509 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:24.750491    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-82h4r" podUID="1b792dc0-5e1c-40c4-a6c6-f68b4c6c8aac"
Nov 03 22:12:24.973511 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:24.973464    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-p4p25" event={"ID":"cc66a3de-7e7e-4855-8f17-db97dcfa815a","Type":"ContainerStarted","Data":"6fb53db817a8c4f26047e3bbfea881a40973b39e7a5c61e7cec3b07187802139"}
Nov 03 22:12:24.982135 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:24.981685    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-73.ec2.internal" event={"ID":"9e2325705fcd5fefcf1f90504938970b","Type":"ContainerStarted","Data":"e52b79bc73f4045b5af1f640cbeb45337d0274daa92a7a674e6962cd198cb6e5"}
Nov 03 22:12:25.008384 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:25.008290    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-p4p25" podStartSLOduration=1.282472303 podStartE2EDuration="7.008270799s" totalImagesPullingTime="5.725798496s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:12:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:25.008052259 +0000 UTC m=+7.719115695" watchObservedRunningTime="2026-11-03 22:12:25.008270799 +0000 UTC m=+7.719334236"
Nov 03 22:12:25.008760 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:25.008725    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-7p8pl" podStartSLOduration=2.41425204 podStartE2EDuration="7.008713236s" totalImagesPullingTime="4.594461196s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:12:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:24.116443326 +0000 UTC m=+6.827506763" watchObservedRunningTime="2026-11-03 22:12:25.008713236 +0000 UTC m=+7.719776675"
Nov 03 22:12:25.755474 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:25.755441    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-66m8v"
Nov 03 22:12:25.755642 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:25.755594    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-66m8v" podUID="8ea0b360-1d72-4d40-935e-1394fecad74c"
Nov 03 22:12:25.986991 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:25.986950    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zcr2" event={"ID":"fa1fd639-c265-4048-b9be-a583885ccb4e","Type":"ContainerStarted","Data":"f6d0cd43f27ab7e2edaa2e46d77238e64a49b9e4f022c295accb53186568fe53"}
Nov 03 22:12:26.409311 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:26.409272    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b071df2-94c8-4459-89f2-f4b448ceb581-metrics-certs\") pod \"network-metrics-daemon-jn5wf\" (UID: \"9b071df2-94c8-4459-89f2-f4b448ceb581\") " pod="openshift-multus/network-metrics-daemon-jn5wf"
Nov 03 22:12:26.409442 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:26.409323    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8ea0b360-1d72-4d40-935e-1394fecad74c-original-pull-secret\") pod \"global-pull-secret-syncer-66m8v\" (UID: \"8ea0b360-1d72-4d40-935e-1394fecad74c\") " pod="kube-system/global-pull-secret-syncer-66m8v"
Nov 03 22:12:26.409514 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:26.409467    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 22:12:26.409569 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:26.409529    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ea0b360-1d72-4d40-935e-1394fecad74c-original-pull-secret podName:8ea0b360-1d72-4d40-935e-1394fecad74c nodeName:}" failed. No retries permitted until 2026-11-03 22:12:34.409511451 +0000 UTC m=+17.120574885 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8ea0b360-1d72-4d40-935e-1394fecad74c-original-pull-secret") pod "global-pull-secret-syncer-66m8v" (UID: "8ea0b360-1d72-4d40-935e-1394fecad74c") : object "kube-system"/"original-pull-secret" not registered
Nov 03 22:12:26.411530 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:26.410366    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 22:12:26.411530 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:26.410417    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b071df2-94c8-4459-89f2-f4b448ceb581-metrics-certs podName:9b071df2-94c8-4459-89f2-f4b448ceb581 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:34.41040556 +0000 UTC m=+17.121468975 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9b071df2-94c8-4459-89f2-f4b448ceb581-metrics-certs") pod "network-metrics-daemon-jn5wf" (UID: "9b071df2-94c8-4459-89f2-f4b448ceb581") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 22:12:26.477482 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:26.477411    2570 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 22:12:26.510910 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:26.510589    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdtpv\" (UniqueName: \"kubernetes.io/projected/1b792dc0-5e1c-40c4-a6c6-f68b4c6c8aac-kube-api-access-qdtpv\") pod \"network-check-target-82h4r\" (UID: \"1b792dc0-5e1c-40c4-a6c6-f68b4c6c8aac\") " pod="openshift-network-diagnostics/network-check-target-82h4r"
Nov 03 22:12:26.510910 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:26.510759    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 22:12:26.510910 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:26.510780    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 22:12:26.510910 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:26.510794    2570 projected.go:196] Error preparing data for projected volume kube-api-access-qdtpv for pod openshift-network-diagnostics/network-check-target-82h4r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 22:12:26.510910 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:26.510849    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b792dc0-5e1c-40c4-a6c6-f68b4c6c8aac-kube-api-access-qdtpv podName:1b792dc0-5e1c-40c4-a6c6-f68b4c6c8aac nodeName:}" failed. No retries permitted until 2026-11-03 22:12:34.510829395 +0000 UTC m=+17.221892825 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-qdtpv" (UniqueName: "kubernetes.io/projected/1b792dc0-5e1c-40c4-a6c6-f68b4c6c8aac-kube-api-access-qdtpv") pod "network-check-target-82h4r" (UID: "1b792dc0-5e1c-40c4-a6c6-f68b4c6c8aac") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 22:12:26.691867 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:26.691003    2570 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T22:12:26.4774338Z","UUID":"73e9f93a-6b2e-4ff3-9f6f-dac06a4b2b48","Handler":null,"Name":"","Endpoint":""}
Nov 03 22:12:26.693662 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:26.693123    2570 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 22:12:26.693662 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:26.693158    2570 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 22:12:26.750449 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:26.749774    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-82h4r"
Nov 03 22:12:26.750449 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:26.749905    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-82h4r" podUID="1b792dc0-5e1c-40c4-a6c6-f68b4c6c8aac"
Nov 03 22:12:26.750449 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:26.750256    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jn5wf"
Nov 03 22:12:26.750449 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:26.750356    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jn5wf" podUID="9b071df2-94c8-4459-89f2-f4b448ceb581"
Nov 03 22:12:26.993793 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:26.993674    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zcr2" event={"ID":"fa1fd639-c265-4048-b9be-a583885ccb4e","Type":"ContainerStarted","Data":"ad74ccf04f3777921bcd76d5caa9580a0494c6ee75bdb38a2d344c227bb8d100"}
Nov 03 22:12:27.752815 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:27.752786    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-66m8v"
Nov 03 22:12:27.752970 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:27.752913    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-66m8v" podUID="8ea0b360-1d72-4d40-935e-1394fecad74c"
Nov 03 22:12:27.997927 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:27.997892    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-x7cpm" event={"ID":"94868020-2e98-4295-8db5-043461cc6e9d","Type":"ContainerStarted","Data":"bee9186a0886dbb497f5a40418c558096b1b5b6ec65d4ff520d9a91e91c268c6"}
Nov 03 22:12:27.999940 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:27.999908    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2k9fs" event={"ID":"ac5ecdf9-3675-4d4f-98e3-a4e8796669be","Type":"ContainerStarted","Data":"2b35cddbfecda1d9959ea03850580b85da1f193827d490814bcff19477762de9"}
Nov 03 22:12:28.002973 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:28.002943    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zcr2" event={"ID":"fa1fd639-c265-4048-b9be-a583885ccb4e","Type":"ContainerStarted","Data":"45d866ad6b7ff1119302f1fe4ac385f2c1e908f713015f8a6f434300bf88d2fc"}
Nov 03 22:12:28.033640 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:28.033581    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-73.ec2.internal" podStartSLOduration=10.033553556 podStartE2EDuration="10.033553556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:12:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:25.045527181 +0000 UTC m=+7.756590618" watchObservedRunningTime="2026-11-03 22:12:28.033553556 +0000 UTC m=+10.744616988"
Nov 03 22:12:28.034244 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:28.034190    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-x7cpm" podStartSLOduration=1.590420795 podStartE2EDuration="10.034175237s" totalImagesPullingTime="8.443754442s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:12:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:28.033268082 +0000 UTC m=+10.744331516" watchObservedRunningTime="2026-11-03 22:12:28.034175237 +0000 UTC m=+10.745238673"
Nov 03 22:12:28.071868 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:28.071818    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-2k9fs" podStartSLOduration=1.9609037219999998 podStartE2EDuration="10.071805179s" totalImagesPullingTime="8.110901457s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:12:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:28.071565628 +0000 UTC m=+10.782629067" watchObservedRunningTime="2026-11-03 22:12:28.071805179 +0000 UTC m=+10.782868641"
Nov 03 22:12:28.111031 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:28.110967    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zcr2" podStartSLOduration=1.905569115 podStartE2EDuration="10.110950011s" totalImagesPullingTime="8.205380896s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:12:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:28.110453082 +0000 UTC m=+10.821516518" watchObservedRunningTime="2026-11-03 22:12:28.110950011 +0000 UTC m=+10.822013448"
Nov 03 22:12:28.731587 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:28.730610    2570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-7p8pl"
Nov 03 22:12:28.732016 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:28.731992    2570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-7p8pl"
Nov 03 22:12:28.750007 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:28.749969    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-82h4r"
Nov 03 22:12:28.750150 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:28.750063    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-82h4r" podUID="1b792dc0-5e1c-40c4-a6c6-f68b4c6c8aac"
Nov 03 22:12:28.750379 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:28.750359    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jn5wf"
Nov 03 22:12:28.750462 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:28.750447    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jn5wf" podUID="9b071df2-94c8-4459-89f2-f4b448ceb581"
Nov 03 22:12:29.009056 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:29.008975    2570 generic.go:356] "Generic (PLEG): container finished" podID="089f9a4b-8eec-4758-811b-e506b41cecbb" containerID="42af517f7c4b4578b80c3649684ffe8c3779ef369b2d3abb6aeb7c5ee2ab53a5" exitCode=0
Nov 03 22:12:29.009714 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:29.009691    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4sm9h" event={"ID":"089f9a4b-8eec-4758-811b-e506b41cecbb","Type":"ContainerDied","Data":"42af517f7c4b4578b80c3649684ffe8c3779ef369b2d3abb6aeb7c5ee2ab53a5"}
Nov 03 22:12:29.750373 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:29.750336    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-66m8v"
Nov 03 22:12:29.750528 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:29.750468    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-66m8v" podUID="8ea0b360-1d72-4d40-935e-1394fecad74c"
Nov 03 22:12:30.014267 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:30.014163    2570 generic.go:356] "Generic (PLEG): container finished" podID="089f9a4b-8eec-4758-811b-e506b41cecbb" containerID="d0ae1b2e3b72d5460f0938d7cd8be1ff2cccac83c4abd7491f80b24e2123188d" exitCode=0
Nov 03 22:12:30.014267 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:30.014258    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4sm9h" event={"ID":"089f9a4b-8eec-4758-811b-e506b41cecbb","Type":"ContainerDied","Data":"d0ae1b2e3b72d5460f0938d7cd8be1ff2cccac83c4abd7491f80b24e2123188d"}
Nov 03 22:12:30.016188 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:30.016141    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7rqgq" event={"ID":"867a2a76-8aa3-4b07-98bf-913c2d2a5562","Type":"ContainerStarted","Data":"e267638bef142f41af944bd8f07ab6e4274a9975bf91d72c8397cc881b28089d"}
Nov 03 22:12:30.018155 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:30.018130    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9ltv9" event={"ID":"a06ec3e9-0a6e-46d8-ade5-13d657a6284c","Type":"ContainerStarted","Data":"65b835854e536f8563a76f4c0f276eb4fc8b5df479fbd30cec1b646ea99c00b9"}
Nov 03 22:12:30.096400 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:30.096339    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-9ltv9" podStartSLOduration=1.804077607 podStartE2EDuration="12.096319853s" totalImagesPullingTime="10.292242246s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:12:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:30.096252218 +0000 UTC m=+12.807315655" watchObservedRunningTime="2026-11-03 22:12:30.096319853 +0000 UTC m=+12.807383287"
Nov 03 22:12:30.131887 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:30.131831    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-7rqgq" podStartSLOduration=3.407428888 podStartE2EDuration="12.131813479s" totalImagesPullingTime="8.724384591s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:12:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:30.131266932 +0000 UTC m=+12.842330374" watchObservedRunningTime="2026-11-03 22:12:30.131813479 +0000 UTC m=+12.842876914"
Nov 03 22:12:30.750031 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:30.750003    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jn5wf"
Nov 03 22:12:30.750194 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:30.750164    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jn5wf" podUID="9b071df2-94c8-4459-89f2-f4b448ceb581"
Nov 03 22:12:30.750262 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:30.750229    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-82h4r"
Nov 03 22:12:30.750354 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:30.750335    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-82h4r" podUID="1b792dc0-5e1c-40c4-a6c6-f68b4c6c8aac"
Nov 03 22:12:31.023033 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:31.022951    2570 generic.go:356] "Generic (PLEG): container finished" podID="089f9a4b-8eec-4758-811b-e506b41cecbb" containerID="640f26bacadb38ff46b56466290f040fc7efba4e04872b3c8309eb8e0a8cd9c4" exitCode=0
Nov 03 22:12:31.023610 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:31.023038    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4sm9h" event={"ID":"089f9a4b-8eec-4758-811b-e506b41cecbb","Type":"ContainerDied","Data":"640f26bacadb38ff46b56466290f040fc7efba4e04872b3c8309eb8e0a8cd9c4"}
Nov 03 22:12:31.750242 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:31.750110    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-66m8v"
Nov 03 22:12:31.750417 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:31.750265    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-66m8v" podUID="8ea0b360-1d72-4d40-935e-1394fecad74c"
Nov 03 22:12:32.750864 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:32.750830    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jn5wf"
Nov 03 22:12:32.751514 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:32.750869    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-82h4r"
Nov 03 22:12:32.751514 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:32.750960    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jn5wf" podUID="9b071df2-94c8-4459-89f2-f4b448ceb581"
Nov 03 22:12:32.751514 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:32.751098    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-82h4r" podUID="1b792dc0-5e1c-40c4-a6c6-f68b4c6c8aac"
Nov 03 22:12:33.030955 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:33.030925    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4sm9h" event={"ID":"089f9a4b-8eec-4758-811b-e506b41cecbb","Type":"ContainerStarted","Data":"9a5f82966fff3bdd089db9bb9ae7b8893c357f728b516ae00dc153d198a006a8"}
Nov 03 22:12:33.750671 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:33.750495    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-66m8v"
Nov 03 22:12:33.750831 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:33.750748    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-66m8v" podUID="8ea0b360-1d72-4d40-935e-1394fecad74c"
Nov 03 22:12:34.035783 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:34.035753    2570 generic.go:356] "Generic (PLEG): container finished" podID="089f9a4b-8eec-4758-811b-e506b41cecbb" containerID="9a5f82966fff3bdd089db9bb9ae7b8893c357f728b516ae00dc153d198a006a8" exitCode=0
Nov 03 22:12:34.036073 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:34.035807    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4sm9h" event={"ID":"089f9a4b-8eec-4758-811b-e506b41cecbb","Type":"ContainerDied","Data":"9a5f82966fff3bdd089db9bb9ae7b8893c357f728b516ae00dc153d198a006a8"}
Nov 03 22:12:34.472384 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:34.472351    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b071df2-94c8-4459-89f2-f4b448ceb581-metrics-certs\") pod \"network-metrics-daemon-jn5wf\" (UID: \"9b071df2-94c8-4459-89f2-f4b448ceb581\") " pod="openshift-multus/network-metrics-daemon-jn5wf"
Nov 03 22:12:34.472542 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:34.472393    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8ea0b360-1d72-4d40-935e-1394fecad74c-original-pull-secret\") pod \"global-pull-secret-syncer-66m8v\" (UID: \"8ea0b360-1d72-4d40-935e-1394fecad74c\") " pod="kube-system/global-pull-secret-syncer-66m8v"
Nov 03 22:12:34.472542 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:34.472508    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 22:12:34.472542 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:34.472515    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 22:12:34.472661 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:34.472554    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ea0b360-1d72-4d40-935e-1394fecad74c-original-pull-secret podName:8ea0b360-1d72-4d40-935e-1394fecad74c nodeName:}" failed. No retries permitted until 2026-11-03 22:12:50.472542769 +0000 UTC m=+33.183606183 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8ea0b360-1d72-4d40-935e-1394fecad74c-original-pull-secret") pod "global-pull-secret-syncer-66m8v" (UID: "8ea0b360-1d72-4d40-935e-1394fecad74c") : object "kube-system"/"original-pull-secret" not registered
Nov 03 22:12:34.472661 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:34.472577    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b071df2-94c8-4459-89f2-f4b448ceb581-metrics-certs podName:9b071df2-94c8-4459-89f2-f4b448ceb581 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:50.472560023 +0000 UTC m=+33.183623438 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9b071df2-94c8-4459-89f2-f4b448ceb581-metrics-certs") pod "network-metrics-daemon-jn5wf" (UID: "9b071df2-94c8-4459-89f2-f4b448ceb581") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 22:12:34.573759 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:34.573680    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdtpv\" (UniqueName: \"kubernetes.io/projected/1b792dc0-5e1c-40c4-a6c6-f68b4c6c8aac-kube-api-access-qdtpv\") pod \"network-check-target-82h4r\" (UID: \"1b792dc0-5e1c-40c4-a6c6-f68b4c6c8aac\") " pod="openshift-network-diagnostics/network-check-target-82h4r"
Nov 03 22:12:34.573890 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:34.573822    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 22:12:34.573890 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:34.573838    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 22:12:34.573890 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:34.573847    2570 projected.go:196] Error preparing data for projected volume kube-api-access-qdtpv for pod openshift-network-diagnostics/network-check-target-82h4r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 22:12:34.573984 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:34.573898    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b792dc0-5e1c-40c4-a6c6-f68b4c6c8aac-kube-api-access-qdtpv podName:1b792dc0-5e1c-40c4-a6c6-f68b4c6c8aac nodeName:}" failed. No retries permitted until 2026-11-03 22:12:50.573883544 +0000 UTC m=+33.284946962 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-qdtpv" (UniqueName: "kubernetes.io/projected/1b792dc0-5e1c-40c4-a6c6-f68b4c6c8aac-kube-api-access-qdtpv") pod "network-check-target-82h4r" (UID: "1b792dc0-5e1c-40c4-a6c6-f68b4c6c8aac") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 22:12:34.750586 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:34.750552    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-82h4r"
Nov 03 22:12:34.750586 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:34.750579    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jn5wf"
Nov 03 22:12:34.750785 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:34.750649    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-82h4r" podUID="1b792dc0-5e1c-40c4-a6c6-f68b4c6c8aac"
Nov 03 22:12:34.750785 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:34.750762    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jn5wf" podUID="9b071df2-94c8-4459-89f2-f4b448ceb581"
Nov 03 22:12:35.040746 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:35.040715    2570 generic.go:356] "Generic (PLEG): container finished" podID="089f9a4b-8eec-4758-811b-e506b41cecbb" containerID="8cac1f1fb9d6b63898381f9e31a4e59ecd28999ea3a9f3c4a9f83a644f7a0051" exitCode=0
Nov 03 22:12:35.041245 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:35.040796    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4sm9h" event={"ID":"089f9a4b-8eec-4758-811b-e506b41cecbb","Type":"ContainerDied","Data":"8cac1f1fb9d6b63898381f9e31a4e59ecd28999ea3a9f3c4a9f83a644f7a0051"}
Nov 03 22:12:35.043606 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:35.043589    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mx5q8_ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea/ovn-acl-logging/0.log"
Nov 03 22:12:35.043970 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:35.043945    2570 generic.go:356] "Generic (PLEG): container finished" podID="ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea" containerID="9a59b13d1f08e177740c394e01192da89fd9050fe180a943a7019dae9e6e4eef" exitCode=1
Nov 03 22:12:35.044039 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:35.043998    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8" event={"ID":"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea","Type":"ContainerStarted","Data":"e35ab74d08ba9164430bc2ce3ab411f7bd086879c8be6c1f42a7c584144547e6"}
Nov 03 22:12:35.044101 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:35.044067    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8" event={"ID":"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea","Type":"ContainerStarted","Data":"ea44f0e8c7ba7a1d892ffa6aae4146aa269fd5fc3d7fc3054c55bc906cde43e3"}
Nov 03 22:12:35.044150 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:35.044119    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8" event={"ID":"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea","Type":"ContainerStarted","Data":"b55f313cbab399821fa8a4a60b1e0303b28ad3f768785c19a646a3667da2c397"}
Nov 03 22:12:35.044198 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:35.044168    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8" event={"ID":"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea","Type":"ContainerStarted","Data":"040512f2d6993c9ec7c0cb49db810bfd239c702c55fd3deeb3601dd3ac24e476"}
Nov 03 22:12:35.044265 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:35.044234    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8" event={"ID":"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea","Type":"ContainerDied","Data":"9a59b13d1f08e177740c394e01192da89fd9050fe180a943a7019dae9e6e4eef"}
Nov 03 22:12:35.044315 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:35.044286    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8" event={"ID":"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea","Type":"ContainerStarted","Data":"2744931af95c29747226f1ea921d7abd39914957e632ef0b4004ce0958d80268"}
Nov 03 22:12:35.750816 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:35.750780    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-66m8v"
Nov 03 22:12:35.750986 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:35.750905    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-66m8v" podUID="8ea0b360-1d72-4d40-935e-1394fecad74c"
Nov 03 22:12:36.048943 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:36.048870    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4sm9h" event={"ID":"089f9a4b-8eec-4758-811b-e506b41cecbb","Type":"ContainerStarted","Data":"4ea58cc3a09d7d8a49c76b6aeb3e28a38a6947cb80b6098bc76f05ec8adfed7d"}
Nov 03 22:12:36.115947 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:36.115905    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-4sm9h" podStartSLOduration=8.184761715 podStartE2EDuration="18.115893373s" totalImagesPullingTime="9.931131658s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:12:18 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:36.115776527 +0000 UTC m=+18.826839963" watchObservedRunningTime="2026-11-03 22:12:36.115893373 +0000 UTC m=+18.826957150"
Nov 03 22:12:36.749932 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:36.749896    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jn5wf"
Nov 03 22:12:36.749932 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:36.749936    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-82h4r"
Nov 03 22:12:36.750113 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:36.750020    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jn5wf" podUID="9b071df2-94c8-4459-89f2-f4b448ceb581"
Nov 03 22:12:36.750146 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:36.750131    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-82h4r" podUID="1b792dc0-5e1c-40c4-a6c6-f68b4c6c8aac"
Nov 03 22:12:37.053520 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:37.053494    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mx5q8_ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea/ovn-acl-logging/0.log"
Nov 03 22:12:37.053924 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:37.053863    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8" event={"ID":"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea","Type":"ContainerStarted","Data":"5cec8948de4a46414b51f753e5faf560973d9090500e5a289df9845023c31ff1"}
Nov 03 22:12:37.750710 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:37.750680    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-66m8v"
Nov 03 22:12:37.750869 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:37.750760    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-66m8v" podUID="8ea0b360-1d72-4d40-935e-1394fecad74c"
Nov 03 22:12:38.750094 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:38.750063    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jn5wf"
Nov 03 22:12:38.750900 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:38.750109    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-82h4r"
Nov 03 22:12:38.750900 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:38.750205    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jn5wf" podUID="9b071df2-94c8-4459-89f2-f4b448ceb581"
Nov 03 22:12:38.750900 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:38.750208    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-82h4r" podUID="1b792dc0-5e1c-40c4-a6c6-f68b4c6c8aac"
Nov 03 22:12:39.061264 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:39.061237    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mx5q8_ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea/ovn-acl-logging/0.log"
Nov 03 22:12:39.061597 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:39.061569    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8" event={"ID":"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea","Type":"ContainerStarted","Data":"900a0bf2aaefb8ba2ec43e7bb70d677eea5a551a3951e1608168d2be9ed7af5b"}
Nov 03 22:12:39.061851 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:39.061835    2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:39.062010 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:39.061993    2570 scope.go:122] "RemoveContainer" containerID="9a59b13d1f08e177740c394e01192da89fd9050fe180a943a7019dae9e6e4eef"
Nov 03 22:12:39.077713 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:39.077692    2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:39.753668 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:39.753486    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-66m8v"
Nov 03 22:12:39.754164 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:39.753741    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-66m8v" podUID="8ea0b360-1d72-4d40-935e-1394fecad74c"
Nov 03 22:12:40.067293 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:40.067206    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mx5q8_ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea/ovn-acl-logging/0.log"
Nov 03 22:12:40.067532 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:40.067509    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8" event={"ID":"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea","Type":"ContainerStarted","Data":"971a366ca8f2c6866924dcb9628613953f198d43e6d528a9f53ff8ea4f9d160e"}
Nov 03 22:12:40.067734 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:40.067717    2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:40.067797 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:40.067787    2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:40.082847 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:40.082826    2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:40.202773 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:40.202719    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8" podStartSLOduration=7.272287606 podStartE2EDuration="22.202705529s" totalImagesPullingTime="14.930417923s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:12:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:40.151597817 +0000 UTC m=+22.862661253" watchObservedRunningTime="2026-11-03 22:12:40.202705529 +0000 UTC m=+22.913768964"
Nov 03 22:12:40.670844 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:40.670812    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-82h4r"]
Nov 03 22:12:40.671040 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:40.670994    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-82h4r"
Nov 03 22:12:40.671143 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:40.671106    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-82h4r" podUID="1b792dc0-5e1c-40c4-a6c6-f68b4c6c8aac"
Nov 03 22:12:40.671512 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:40.671483    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-66m8v"]
Nov 03 22:12:40.671636 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:40.671593    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-66m8v"
Nov 03 22:12:40.671683 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:40.671671    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-66m8v" podUID="8ea0b360-1d72-4d40-935e-1394fecad74c"
Nov 03 22:12:40.674140 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:40.674118    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jn5wf"]
Nov 03 22:12:40.674255 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:40.674244    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jn5wf"
Nov 03 22:12:40.674366 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:40.674349    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jn5wf" podUID="9b071df2-94c8-4459-89f2-f4b448ceb581"
Nov 03 22:12:42.750052 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:42.750019    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-66m8v"
Nov 03 22:12:42.750052 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:42.750039    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jn5wf"
Nov 03 22:12:42.750052 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:42.750033    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-82h4r"
Nov 03 22:12:42.750567 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:42.750148    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jn5wf" podUID="9b071df2-94c8-4459-89f2-f4b448ceb581"
Nov 03 22:12:42.750567 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:42.750210    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-82h4r" podUID="1b792dc0-5e1c-40c4-a6c6-f68b4c6c8aac"
Nov 03 22:12:42.750567 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:42.750279    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-66m8v" podUID="8ea0b360-1d72-4d40-935e-1394fecad74c"
Nov 03 22:12:44.646506 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:44.646466    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:12:44.750436 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:44.750406    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-66m8v"
Nov 03 22:12:44.750436 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:44.750423    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-82h4r"
Nov 03 22:12:44.750436 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:44.750434    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jn5wf"
Nov 03 22:12:44.750650 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:44.750510    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-66m8v" podUID="8ea0b360-1d72-4d40-935e-1394fecad74c"
Nov 03 22:12:44.750650 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:44.750561    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-82h4r" podUID="1b792dc0-5e1c-40c4-a6c6-f68b4c6c8aac"
Nov 03 22:12:44.750650 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:44.750634    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jn5wf" podUID="9b071df2-94c8-4459-89f2-f4b448ceb581"
Nov 03 22:12:44.880348 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:44.880315    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:12:46.436880 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:46.436850    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:12:46.637671 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:46.637584    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:12:46.749922 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:46.749884    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jn5wf"
Nov 03 22:12:46.750070 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:46.749885    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-82h4r"
Nov 03 22:12:46.750070 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:46.750002    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jn5wf" podUID="9b071df2-94c8-4459-89f2-f4b448ceb581"
Nov 03 22:12:46.750145 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:46.749885    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-66m8v"
Nov 03 22:12:46.750145 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:46.750127    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-66m8v" podUID="8ea0b360-1d72-4d40-935e-1394fecad74c"
Nov 03 22:12:46.750145 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:46.750066    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-82h4r" podUID="1b792dc0-5e1c-40c4-a6c6-f68b4c6c8aac"
Nov 03 22:12:46.769958 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:46.769936    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:12:47.075588 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:47.075561    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-73.ec2.internal" event="NodeReady"
Nov 03 22:12:47.075767 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:47.075662    2570 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 03 22:12:47.136714 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:47.136676    2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-4qrls"]
Nov 03 22:12:47.139428 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:47.139398    2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-vhccw"]
Nov 03 22:12:47.139559 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:47.139506    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4qrls"
Nov 03 22:12:47.143326 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:47.143308    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vhccw"
Nov 03 22:12:47.152584 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:47.152566    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 22:12:47.152673 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:47.152585    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-h6vzq\""
Nov 03 22:12:47.152853 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:47.152836    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 22:12:47.158288 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:47.158270    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 22:12:47.164027 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:47.163999    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4qrls"]
Nov 03 22:12:47.164541 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:47.164523    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vhccw"]
Nov 03 22:12:47.236836 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:47.236804    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:12:47.252696 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:47.252668    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 22:12:47.252824 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:47.252714    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-svbdj\""
Nov 03 22:12:47.253133 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:47.253117    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 22:12:47.267428 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:47.267393    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/365a1938-752c-4cec-a917-c5a46c719307-cert\") pod \"ingress-canary-vhccw\" (UID: \"365a1938-752c-4cec-a917-c5a46c719307\") " pod="openshift-ingress-canary/ingress-canary-vhccw"
Nov 03 22:12:47.267428 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:47.267429    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/309c2503-3fef-4048-af01-6b3b70aa52a1-config-volume\") pod \"dns-default-4qrls\" (UID: \"309c2503-3fef-4048-af01-6b3b70aa52a1\") " pod="openshift-dns/dns-default-4qrls"
Nov 03 22:12:47.267576 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:47.267454    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdv4k\" (UniqueName: \"kubernetes.io/projected/365a1938-752c-4cec-a917-c5a46c719307-kube-api-access-qdv4k\") pod \"ingress-canary-vhccw\" (UID: \"365a1938-752c-4cec-a917-c5a46c719307\") " pod="openshift-ingress-canary/ingress-canary-vhccw"
Nov 03 22:12:47.267576 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:47.267566    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/309c2503-3fef-4048-af01-6b3b70aa52a1-metrics-tls\") pod \"dns-default-4qrls\" (UID: \"309c2503-3fef-4048-af01-6b3b70aa52a1\") " pod="openshift-dns/dns-default-4qrls"
Nov 03 22:12:47.267683 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:47.267666    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/309c2503-3fef-4048-af01-6b3b70aa52a1-tmp-dir\") pod \"dns-default-4qrls\" (UID: \"309c2503-3fef-4048-af01-6b3b70aa52a1\") " pod="openshift-dns/dns-default-4qrls"
Nov 03 22:12:47.267723 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:47.267694    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8slxn\" (UniqueName: \"kubernetes.io/projected/309c2503-3fef-4048-af01-6b3b70aa52a1-kube-api-access-8slxn\") pod \"dns-default-4qrls\" (UID: \"309c2503-3fef-4048-af01-6b3b70aa52a1\") " pod="openshift-dns/dns-default-4qrls"
Nov 03 22:12:47.368243 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:47.368152    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/365a1938-752c-4cec-a917-c5a46c719307-cert\") pod \"ingress-canary-vhccw\" (UID: \"365a1938-752c-4cec-a917-c5a46c719307\") " pod="openshift-ingress-canary/ingress-canary-vhccw"
Nov 03 22:12:47.368243 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:47.368188    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/309c2503-3fef-4048-af01-6b3b70aa52a1-config-volume\") pod \"dns-default-4qrls\" (UID: \"309c2503-3fef-4048-af01-6b3b70aa52a1\") " pod="openshift-dns/dns-default-4qrls"
Nov 03 22:12:47.368243 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:47.368210    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdv4k\" (UniqueName: \"kubernetes.io/projected/365a1938-752c-4cec-a917-c5a46c719307-kube-api-access-qdv4k\") pod \"ingress-canary-vhccw\" (UID: \"365a1938-752c-4cec-a917-c5a46c719307\") " pod="openshift-ingress-canary/ingress-canary-vhccw"
Nov 03 22:12:47.368496 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:47.368260    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/309c2503-3fef-4048-af01-6b3b70aa52a1-metrics-tls\") pod \"dns-default-4qrls\" (UID: \"309c2503-3fef-4048-af01-6b3b70aa52a1\") " pod="openshift-dns/dns-default-4qrls"
Nov 03 22:12:47.368496 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:47.368328    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/309c2503-3fef-4048-af01-6b3b70aa52a1-tmp-dir\") pod \"dns-default-4qrls\" (UID: \"309c2503-3fef-4048-af01-6b3b70aa52a1\") " pod="openshift-dns/dns-default-4qrls"
Nov 03 22:12:47.368496 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:47.368352    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8slxn\" (UniqueName: \"kubernetes.io/projected/309c2503-3fef-4048-af01-6b3b70aa52a1-kube-api-access-8slxn\") pod \"dns-default-4qrls\" (UID: \"309c2503-3fef-4048-af01-6b3b70aa52a1\") " pod="openshift-dns/dns-default-4qrls"
Nov 03 22:12:47.368496 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:47.368409    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 22:12:47.368887 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:47.368761    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/309c2503-3fef-4048-af01-6b3b70aa52a1-metrics-tls podName:309c2503-3fef-4048-af01-6b3b70aa52a1 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:47.868620331 +0000 UTC m=+30.579683762 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/309c2503-3fef-4048-af01-6b3b70aa52a1-metrics-tls") pod "dns-default-4qrls" (UID: "309c2503-3fef-4048-af01-6b3b70aa52a1") : secret "dns-default-metrics-tls" not found
Nov 03 22:12:47.369033 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:47.369009    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/309c2503-3fef-4048-af01-6b3b70aa52a1-tmp-dir\") pod \"dns-default-4qrls\" (UID: \"309c2503-3fef-4048-af01-6b3b70aa52a1\") " pod="openshift-dns/dns-default-4qrls"
Nov 03 22:12:47.369472 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:47.369442    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/309c2503-3fef-4048-af01-6b3b70aa52a1-config-volume\") pod \"dns-default-4qrls\" (UID: \"309c2503-3fef-4048-af01-6b3b70aa52a1\") " pod="openshift-dns/dns-default-4qrls"
Nov 03 22:12:47.373113 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:47.373090    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/365a1938-752c-4cec-a917-c5a46c719307-cert\") pod \"ingress-canary-vhccw\" (UID: \"365a1938-752c-4cec-a917-c5a46c719307\") " pod="openshift-ingress-canary/ingress-canary-vhccw"
Nov 03 22:12:47.385357 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:47.385324    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:12:47.390788 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:47.390764    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8slxn\" (UniqueName: \"kubernetes.io/projected/309c2503-3fef-4048-af01-6b3b70aa52a1-kube-api-access-8slxn\") pod \"dns-default-4qrls\" (UID: \"309c2503-3fef-4048-af01-6b3b70aa52a1\") " pod="openshift-dns/dns-default-4qrls"
Nov 03 22:12:47.391107 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:47.391089    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdv4k\" (UniqueName: \"kubernetes.io/projected/365a1938-752c-4cec-a917-c5a46c719307-kube-api-access-qdv4k\") pod \"ingress-canary-vhccw\" (UID: \"365a1938-752c-4cec-a917-c5a46c719307\") " pod="openshift-ingress-canary/ingress-canary-vhccw"
Nov 03 22:12:47.460183 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:47.460152    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vhccw"
Nov 03 22:12:47.767633 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:47.767595    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vhccw"]
Nov 03 22:12:47.771489 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:47.771462    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod365a1938_752c_4cec_a917_c5a46c719307.slice/crio-73a63c22ffae1c6c4abf49a45190081732a2518699b857e09e444501a9ed590a WatchSource:0}: Error finding container 73a63c22ffae1c6c4abf49a45190081732a2518699b857e09e444501a9ed590a: Status 404 returned error can't find the container with id 73a63c22ffae1c6c4abf49a45190081732a2518699b857e09e444501a9ed590a
Nov 03 22:12:47.872058 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:47.872018    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/309c2503-3fef-4048-af01-6b3b70aa52a1-metrics-tls\") pod \"dns-default-4qrls\" (UID: \"309c2503-3fef-4048-af01-6b3b70aa52a1\") " pod="openshift-dns/dns-default-4qrls"
Nov 03 22:12:47.872193 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:47.872133    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 22:12:47.872256 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:47.872206    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/309c2503-3fef-4048-af01-6b3b70aa52a1-metrics-tls podName:309c2503-3fef-4048-af01-6b3b70aa52a1 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:48.872190528 +0000 UTC m=+31.583253947 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/309c2503-3fef-4048-af01-6b3b70aa52a1-metrics-tls") pod "dns-default-4qrls" (UID: "309c2503-3fef-4048-af01-6b3b70aa52a1") : secret "dns-default-metrics-tls" not found
Nov 03 22:12:48.084879 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:48.084808    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vhccw" event={"ID":"365a1938-752c-4cec-a917-c5a46c719307","Type":"ContainerStarted","Data":"73a63c22ffae1c6c4abf49a45190081732a2518699b857e09e444501a9ed590a"}
Nov 03 22:12:48.245229 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:48.245178    2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-7p8pl"
Nov 03 22:12:48.245392 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:48.245334    2570 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 22:12:48.245855 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:48.245831    2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-7p8pl"
Nov 03 22:12:48.269865 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:48.269835    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:12:48.408531 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:48.408441    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:12:48.700967 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:48.700938    2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-q6ndx"]
Nov 03 22:12:48.714513 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:48.714486    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-q6ndx"
Nov 03 22:12:48.725622 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:48.725012    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-q6ndx"]
Nov 03 22:12:48.749840 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:48.749808    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-82h4r"
Nov 03 22:12:48.749991 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:48.749886    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-66m8v"
Nov 03 22:12:48.749991 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:48.749974    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jn5wf"
Nov 03 22:12:48.823884 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:48.823857    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 22:12:48.823884 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:48.823855    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 22:12:48.824381 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:48.824350    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-9wkj9\""
Nov 03 22:12:48.824483 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:48.824380    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-zwzw5\""
Nov 03 22:12:48.824483 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:48.824400    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-mq5fv\""
Nov 03 22:12:48.824483 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:48.824412    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 22:12:48.824483 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:48.824461    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 22:12:48.824483 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:48.824476    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 22:12:48.824660 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:48.824558    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 22:12:48.824660 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:48.824558    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 22:12:48.824660 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:48.824588    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 22:12:48.824660 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:48.824615    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zdncz\""
Nov 03 22:12:48.837208 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:48.837180    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:12:48.880207 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:48.880177    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/309c2503-3fef-4048-af01-6b3b70aa52a1-metrics-tls\") pod \"dns-default-4qrls\" (UID: \"309c2503-3fef-4048-af01-6b3b70aa52a1\") " pod="openshift-dns/dns-default-4qrls"
Nov 03 22:12:48.880378 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:48.880252    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6813e008-7728-4466-979d-3ba72892a647-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q6ndx\" (UID: \"6813e008-7728-4466-979d-3ba72892a647\") " pod="openshift-insights/insights-runtime-extractor-q6ndx"
Nov 03 22:12:48.880378 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:48.880293    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6813e008-7728-4466-979d-3ba72892a647-crio-socket\") pod \"insights-runtime-extractor-q6ndx\" (UID: \"6813e008-7728-4466-979d-3ba72892a647\") " pod="openshift-insights/insights-runtime-extractor-q6ndx"
Nov 03 22:12:48.880378 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:48.880315    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6813e008-7728-4466-979d-3ba72892a647-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q6ndx\" (UID: \"6813e008-7728-4466-979d-3ba72892a647\") " pod="openshift-insights/insights-runtime-extractor-q6ndx"
Nov 03 22:12:48.880378 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:48.880358    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 22:12:48.880546 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:48.880391    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6813e008-7728-4466-979d-3ba72892a647-data-volume\") pod \"insights-runtime-extractor-q6ndx\" (UID: \"6813e008-7728-4466-979d-3ba72892a647\") " pod="openshift-insights/insights-runtime-extractor-q6ndx"
Nov 03 22:12:48.880546 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:48.880415    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/309c2503-3fef-4048-af01-6b3b70aa52a1-metrics-tls podName:309c2503-3fef-4048-af01-6b3b70aa52a1 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:50.880400065 +0000 UTC m=+33.591463479 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/309c2503-3fef-4048-af01-6b3b70aa52a1-metrics-tls") pod "dns-default-4qrls" (UID: "309c2503-3fef-4048-af01-6b3b70aa52a1") : secret "dns-default-metrics-tls" not found
Nov 03 22:12:48.880546 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:48.880477    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6l77\" (UniqueName: \"kubernetes.io/projected/6813e008-7728-4466-979d-3ba72892a647-kube-api-access-b6l77\") pod \"insights-runtime-extractor-q6ndx\" (UID: \"6813e008-7728-4466-979d-3ba72892a647\") " pod="openshift-insights/insights-runtime-extractor-q6ndx"
Nov 03 22:12:48.981367 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:48.981287    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6813e008-7728-4466-979d-3ba72892a647-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q6ndx\" (UID: \"6813e008-7728-4466-979d-3ba72892a647\") " pod="openshift-insights/insights-runtime-extractor-q6ndx"
Nov 03 22:12:48.981367 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:48.981348    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6813e008-7728-4466-979d-3ba72892a647-crio-socket\") pod \"insights-runtime-extractor-q6ndx\" (UID: \"6813e008-7728-4466-979d-3ba72892a647\") " pod="openshift-insights/insights-runtime-extractor-q6ndx"
Nov 03 22:12:48.981591 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:48.981374    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6813e008-7728-4466-979d-3ba72892a647-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q6ndx\" (UID: \"6813e008-7728-4466-979d-3ba72892a647\") " pod="openshift-insights/insights-runtime-extractor-q6ndx"
Nov 03 22:12:48.981591 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:48.981421    2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 22:12:48.981591 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:48.981435    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6813e008-7728-4466-979d-3ba72892a647-data-volume\") pod \"insights-runtime-extractor-q6ndx\" (UID: \"6813e008-7728-4466-979d-3ba72892a647\") " pod="openshift-insights/insights-runtime-extractor-q6ndx"
Nov 03 22:12:48.981591 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:48.981465    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6l77\" (UniqueName: \"kubernetes.io/projected/6813e008-7728-4466-979d-3ba72892a647-kube-api-access-b6l77\") pod \"insights-runtime-extractor-q6ndx\" (UID: \"6813e008-7728-4466-979d-3ba72892a647\") " pod="openshift-insights/insights-runtime-extractor-q6ndx"
Nov 03 22:12:48.981591 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:48.981516    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6813e008-7728-4466-979d-3ba72892a647-kube-rbac-proxy-cm podName:6813e008-7728-4466-979d-3ba72892a647 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:49.481497728 +0000 UTC m=+32.192561142 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6813e008-7728-4466-979d-3ba72892a647-kube-rbac-proxy-cm") pod "insights-runtime-extractor-q6ndx" (UID: "6813e008-7728-4466-979d-3ba72892a647") : configmap "kube-rbac-proxy" not found
Nov 03 22:12:48.981591 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:48.981546    2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 22:12:48.981872 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:48.981605    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6813e008-7728-4466-979d-3ba72892a647-insights-runtime-extractor-tls podName:6813e008-7728-4466-979d-3ba72892a647 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:49.481588697 +0000 UTC m=+32.192652124 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6813e008-7728-4466-979d-3ba72892a647-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q6ndx" (UID: "6813e008-7728-4466-979d-3ba72892a647") : secret "insights-runtime-extractor-tls" not found
Nov 03 22:12:48.981872 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:48.981612    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6813e008-7728-4466-979d-3ba72892a647-crio-socket\") pod \"insights-runtime-extractor-q6ndx\" (UID: \"6813e008-7728-4466-979d-3ba72892a647\") " pod="openshift-insights/insights-runtime-extractor-q6ndx"
Nov 03 22:12:48.981959 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:48.981882    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6813e008-7728-4466-979d-3ba72892a647-data-volume\") pod \"insights-runtime-extractor-q6ndx\" (UID: \"6813e008-7728-4466-979d-3ba72892a647\") " pod="openshift-insights/insights-runtime-extractor-q6ndx"
Nov 03 22:12:49.007359 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:49.007334    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6l77\" (UniqueName: \"kubernetes.io/projected/6813e008-7728-4466-979d-3ba72892a647-kube-api-access-b6l77\") pod \"insights-runtime-extractor-q6ndx\" (UID: \"6813e008-7728-4466-979d-3ba72892a647\") " pod="openshift-insights/insights-runtime-extractor-q6ndx"
Nov 03 22:12:49.041512 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:49.041481    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:12:49.241501 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:49.241419    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:12:49.392083 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:49.392045    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:12:49.485547 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:49.485512    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6813e008-7728-4466-979d-3ba72892a647-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q6ndx\" (UID: \"6813e008-7728-4466-979d-3ba72892a647\") " pod="openshift-insights/insights-runtime-extractor-q6ndx"
Nov 03 22:12:49.485728 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:49.485565    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6813e008-7728-4466-979d-3ba72892a647-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q6ndx\" (UID: \"6813e008-7728-4466-979d-3ba72892a647\") " pod="openshift-insights/insights-runtime-extractor-q6ndx"
Nov 03 22:12:49.485728 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:49.485652    2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 22:12:49.485728 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:49.485679    2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 22:12:49.485902 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:49.485730    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6813e008-7728-4466-979d-3ba72892a647-kube-rbac-proxy-cm podName:6813e008-7728-4466-979d-3ba72892a647 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:50.485710982 +0000 UTC m=+33.196774407 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6813e008-7728-4466-979d-3ba72892a647-kube-rbac-proxy-cm") pod "insights-runtime-extractor-q6ndx" (UID: "6813e008-7728-4466-979d-3ba72892a647") : configmap "kube-rbac-proxy" not found
Nov 03 22:12:49.485902 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:49.485751    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6813e008-7728-4466-979d-3ba72892a647-insights-runtime-extractor-tls podName:6813e008-7728-4466-979d-3ba72892a647 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:50.485741974 +0000 UTC m=+33.196805391 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6813e008-7728-4466-979d-3ba72892a647-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q6ndx" (UID: "6813e008-7728-4466-979d-3ba72892a647") : secret "insights-runtime-extractor-tls" not found
Nov 03 22:12:50.140368 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:50.140322    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:12:50.148965 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:50.148935    2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-8dbwq"]
Nov 03 22:12:50.169503 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:50.169477    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8dbwq"
Nov 03 22:12:50.172606 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:50.172584    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-8dbwq"]
Nov 03 22:12:50.183078 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:50.183060    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 03 22:12:50.193172 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:50.193154    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 03 22:12:50.193381 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:50.193366    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-sjknz\""
Nov 03 22:12:50.290710 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:50.290680    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgx8f\" (UniqueName: \"kubernetes.io/projected/6498eed3-6528-4a49-a5d6-3dfcf9d75017-kube-api-access-xgx8f\") pod \"migrator-66df5d496b-8dbwq\" (UID: \"6498eed3-6528-4a49-a5d6-3dfcf9d75017\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8dbwq"
Nov 03 22:12:50.291409 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:50.291387    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:12:50.391327 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:50.391245    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgx8f\" (UniqueName: \"kubernetes.io/projected/6498eed3-6528-4a49-a5d6-3dfcf9d75017-kube-api-access-xgx8f\") pod \"migrator-66df5d496b-8dbwq\" (UID: \"6498eed3-6528-4a49-a5d6-3dfcf9d75017\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8dbwq"
Nov 03 22:12:50.416454 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:50.416427    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgx8f\" (UniqueName: \"kubernetes.io/projected/6498eed3-6528-4a49-a5d6-3dfcf9d75017-kube-api-access-xgx8f\") pod \"migrator-66df5d496b-8dbwq\" (UID: \"6498eed3-6528-4a49-a5d6-3dfcf9d75017\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8dbwq"
Nov 03 22:12:50.464368 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:50.464333    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:12:50.490814 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:50.490788    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8dbwq"
Nov 03 22:12:50.491980 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:50.491949    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b071df2-94c8-4459-89f2-f4b448ceb581-metrics-certs\") pod \"network-metrics-daemon-jn5wf\" (UID: \"9b071df2-94c8-4459-89f2-f4b448ceb581\") " pod="openshift-multus/network-metrics-daemon-jn5wf"
Nov 03 22:12:50.492103 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:50.491990    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6813e008-7728-4466-979d-3ba72892a647-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q6ndx\" (UID: \"6813e008-7728-4466-979d-3ba72892a647\") " pod="openshift-insights/insights-runtime-extractor-q6ndx"
Nov 03 22:12:50.492103 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:50.492022    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8ea0b360-1d72-4d40-935e-1394fecad74c-original-pull-secret\") pod \"global-pull-secret-syncer-66m8v\" (UID: \"8ea0b360-1d72-4d40-935e-1394fecad74c\") " pod="kube-system/global-pull-secret-syncer-66m8v"
Nov 03 22:12:50.492103 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:50.492048    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6813e008-7728-4466-979d-3ba72892a647-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q6ndx\" (UID: \"6813e008-7728-4466-979d-3ba72892a647\") " pod="openshift-insights/insights-runtime-extractor-q6ndx"
Nov 03 22:12:50.492103 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:50.492093    2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 22:12:50.492326 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:50.492160    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6813e008-7728-4466-979d-3ba72892a647-kube-rbac-proxy-cm podName:6813e008-7728-4466-979d-3ba72892a647 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:52.492140727 +0000 UTC m=+35.203204143 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6813e008-7728-4466-979d-3ba72892a647-kube-rbac-proxy-cm") pod "insights-runtime-extractor-q6ndx" (UID: "6813e008-7728-4466-979d-3ba72892a647") : configmap "kube-rbac-proxy" not found
Nov 03 22:12:50.492326 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:50.492100    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 03 22:12:50.492326 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:50.492160    2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 22:12:50.492326 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:50.492247    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b071df2-94c8-4459-89f2-f4b448ceb581-metrics-certs podName:9b071df2-94c8-4459-89f2-f4b448ceb581 nodeName:}" failed. No retries permitted until 2026-11-03 22:13:22.492226468 +0000 UTC m=+65.203289898 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9b071df2-94c8-4459-89f2-f4b448ceb581-metrics-certs") pod "network-metrics-daemon-jn5wf" (UID: "9b071df2-94c8-4459-89f2-f4b448ceb581") : secret "metrics-daemon-secret" not found
Nov 03 22:12:50.492326 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:50.492268    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6813e008-7728-4466-979d-3ba72892a647-insights-runtime-extractor-tls podName:6813e008-7728-4466-979d-3ba72892a647 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:52.492257888 +0000 UTC m=+35.203321319 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6813e008-7728-4466-979d-3ba72892a647-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q6ndx" (UID: "6813e008-7728-4466-979d-3ba72892a647") : secret "insights-runtime-extractor-tls" not found
Nov 03 22:12:50.494312 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:50.494293    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8ea0b360-1d72-4d40-935e-1394fecad74c-original-pull-secret\") pod \"global-pull-secret-syncer-66m8v\" (UID: \"8ea0b360-1d72-4d40-935e-1394fecad74c\") " pod="kube-system/global-pull-secret-syncer-66m8v"
Nov 03 22:12:50.592678 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:50.592643    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdtpv\" (UniqueName: \"kubernetes.io/projected/1b792dc0-5e1c-40c4-a6c6-f68b4c6c8aac-kube-api-access-qdtpv\") pod \"network-check-target-82h4r\" (UID: \"1b792dc0-5e1c-40c4-a6c6-f68b4c6c8aac\") " pod="openshift-network-diagnostics/network-check-target-82h4r"
Nov 03 22:12:50.594946 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:50.594925    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdtpv\" (UniqueName: \"kubernetes.io/projected/1b792dc0-5e1c-40c4-a6c6-f68b4c6c8aac-kube-api-access-qdtpv\") pod \"network-check-target-82h4r\" (UID: \"1b792dc0-5e1c-40c4-a6c6-f68b4c6c8aac\") " pod="openshift-network-diagnostics/network-check-target-82h4r"
Nov 03 22:12:50.629191 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:50.629149    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-66m8v"
Nov 03 22:12:50.794577 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:50.794546    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-8dbwq"]
Nov 03 22:12:50.797341 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:50.797317    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6498eed3_6528_4a49_a5d6_3dfcf9d75017.slice/crio-537a5c4de9dae0664356e0290f054d99f3eefc22cc0b7682faa7cd0146fa2da5 WatchSource:0}: Error finding container 537a5c4de9dae0664356e0290f054d99f3eefc22cc0b7682faa7cd0146fa2da5: Status 404 returned error can't find the container with id 537a5c4de9dae0664356e0290f054d99f3eefc22cc0b7682faa7cd0146fa2da5
Nov 03 22:12:50.874758 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:50.874722    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:12:50.886617 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:50.886593    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-82h4r"
Nov 03 22:12:50.894956 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:50.894892    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/309c2503-3fef-4048-af01-6b3b70aa52a1-metrics-tls\") pod \"dns-default-4qrls\" (UID: \"309c2503-3fef-4048-af01-6b3b70aa52a1\") " pod="openshift-dns/dns-default-4qrls"
Nov 03 22:12:50.895057 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:50.895022    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 22:12:50.895122 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:50.895089    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/309c2503-3fef-4048-af01-6b3b70aa52a1-metrics-tls podName:309c2503-3fef-4048-af01-6b3b70aa52a1 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:54.895068543 +0000 UTC m=+37.606131974 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/309c2503-3fef-4048-af01-6b3b70aa52a1-metrics-tls") pod "dns-default-4qrls" (UID: "309c2503-3fef-4048-af01-6b3b70aa52a1") : secret "dns-default-metrics-tls" not found
Nov 03 22:12:50.934092 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:50.934059    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-66m8v"]
Nov 03 22:12:50.937809 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:50.937784    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ea0b360_1d72_4d40_935e_1394fecad74c.slice/crio-0452dadfd2655386a337e6ef379ca20341938653d5f53aa2545e56566fe7e56a WatchSource:0}: Error finding container 0452dadfd2655386a337e6ef379ca20341938653d5f53aa2545e56566fe7e56a: Status 404 returned error can't find the container with id 0452dadfd2655386a337e6ef379ca20341938653d5f53aa2545e56566fe7e56a
Nov 03 22:12:51.093501 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:51.093468    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-66m8v" event={"ID":"8ea0b360-1d72-4d40-935e-1394fecad74c","Type":"ContainerStarted","Data":"0452dadfd2655386a337e6ef379ca20341938653d5f53aa2545e56566fe7e56a"}
Nov 03 22:12:51.095037 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:51.095013    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vhccw" event={"ID":"365a1938-752c-4cec-a917-c5a46c719307","Type":"ContainerStarted","Data":"96d106415309952aa2bdc5f70f8088e898fcd11b63871ee26157d17a775002b9"}
Nov 03 22:12:51.096230 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:51.096191    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8dbwq" event={"ID":"6498eed3-6528-4a49-a5d6-3dfcf9d75017","Type":"ContainerStarted","Data":"537a5c4de9dae0664356e0290f054d99f3eefc22cc0b7682faa7cd0146fa2da5"}
Nov 03 22:12:51.131728 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:51.131677    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-vhccw" podStartSLOduration=1.5108337010000001 podStartE2EDuration="4.131661138s" totalImagesPullingTime="2.620827437s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:12:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:51.131542095 +0000 UTC m=+33.842605532" watchObservedRunningTime="2026-11-03 22:12:51.131661138 +0000 UTC m=+33.842724599"
Nov 03 22:12:51.207885 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:51.207854    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-82h4r"]
Nov 03 22:12:51.210457 ip-10-0-139-73 kubenswrapper[2570]: W1103 22:12:51.210432    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b792dc0_5e1c_40c4_a6c6_f68b4c6c8aac.slice/crio-cbd1adc5425c7bbbc9dbc85c8ea9f67b3383c6657e7b8dc4ed932383257eed4a WatchSource:0}: Error finding container cbd1adc5425c7bbbc9dbc85c8ea9f67b3383c6657e7b8dc4ed932383257eed4a: Status 404 returned error can't find the container with id cbd1adc5425c7bbbc9dbc85c8ea9f67b3383c6657e7b8dc4ed932383257eed4a
Nov 03 22:12:52.100859 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:52.100745    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-82h4r" event={"ID":"1b792dc0-5e1c-40c4-a6c6-f68b4c6c8aac","Type":"ContainerStarted","Data":"cbd1adc5425c7bbbc9dbc85c8ea9f67b3383c6657e7b8dc4ed932383257eed4a"}
Nov 03 22:12:52.508551 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:52.508516    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6813e008-7728-4466-979d-3ba72892a647-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q6ndx\" (UID: \"6813e008-7728-4466-979d-3ba72892a647\") " pod="openshift-insights/insights-runtime-extractor-q6ndx"
Nov 03 22:12:52.509050 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:52.508613    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6813e008-7728-4466-979d-3ba72892a647-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q6ndx\" (UID: \"6813e008-7728-4466-979d-3ba72892a647\") " pod="openshift-insights/insights-runtime-extractor-q6ndx"
Nov 03 22:12:52.509050 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:52.508697    2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 22:12:52.509050 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:52.508748    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6813e008-7728-4466-979d-3ba72892a647-kube-rbac-proxy-cm podName:6813e008-7728-4466-979d-3ba72892a647 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:56.508736437 +0000 UTC m=+39.219799864 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6813e008-7728-4466-979d-3ba72892a647-kube-rbac-proxy-cm") pod "insights-runtime-extractor-q6ndx" (UID: "6813e008-7728-4466-979d-3ba72892a647") : configmap "kube-rbac-proxy" not found
Nov 03 22:12:52.509242 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:52.509098    2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 22:12:52.509242 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:52.509151    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6813e008-7728-4466-979d-3ba72892a647-insights-runtime-extractor-tls podName:6813e008-7728-4466-979d-3ba72892a647 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:56.509133767 +0000 UTC m=+39.220197196 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6813e008-7728-4466-979d-3ba72892a647-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q6ndx" (UID: "6813e008-7728-4466-979d-3ba72892a647") : secret "insights-runtime-extractor-tls" not found
Nov 03 22:12:53.105603 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:53.105560    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8dbwq" event={"ID":"6498eed3-6528-4a49-a5d6-3dfcf9d75017","Type":"ContainerStarted","Data":"2f8391dec768bef0cd2d73a1417a313100ef7c0374ae0db2b2d3ae8ce88855f5"}
Nov 03 22:12:53.144302 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:53.144233    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8dbwq" podStartSLOduration=1.665773151 podStartE2EDuration="3.144202693s" totalImagesPullingTime="1.478429542s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:12:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:53.143444363 +0000 UTC m=+35.854507797" watchObservedRunningTime="2026-11-03 22:12:53.144202693 +0000 UTC m=+35.855266123"
Nov 03 22:12:54.205508 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:54.205467    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:12:54.464703 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:54.464627    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:12:54.633984 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:54.633949    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:12:54.770079 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:54.770051    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:12:54.929581 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:54.929542    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/309c2503-3fef-4048-af01-6b3b70aa52a1-metrics-tls\") pod \"dns-default-4qrls\" (UID: \"309c2503-3fef-4048-af01-6b3b70aa52a1\") " pod="openshift-dns/dns-default-4qrls"
Nov 03 22:12:54.929748 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:54.929672    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 22:12:54.929748 ip-10-0-139-73 kubenswrapper[2570]: E1103 22:12:54.929727    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/309c2503-3fef-4048-af01-6b3b70aa52a1-metrics-tls podName:309c2503-3fef-4048-af01-6b3b70aa52a1 nodeName:}" failed. No retries permitted until 2026-11-03 22:13:02.929711901 +0000 UTC m=+45.640775315 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/309c2503-3fef-4048-af01-6b3b70aa52a1-metrics-tls") pod "dns-default-4qrls" (UID: "309c2503-3fef-4048-af01-6b3b70aa52a1") : secret "dns-default-metrics-tls" not found
Nov 03 22:12:54.957839 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:54.957808    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:12:55.102052 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:55.101977    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:12:55.112626 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:55.112595    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-82h4r" event={"ID":"1b792dc0-5e1c-40c4-a6c6-f68b4c6c8aac","Type":"ContainerStarted","Data":"ce4dfaf2ed46b7ea556b0fd7dd76ac7d19d906e742a227a7a4d9d6e6b1ebf12a"}
Nov 03 22:12:55.112860 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:55.112845    2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-82h4r"
Nov 03 22:12:55.151704 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:55.151655    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-82h4r" podStartSLOduration=33.67511937 podStartE2EDuration="37.151641594s" totalImagesPullingTime="3.476522224s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:12:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:55.151171771 +0000 UTC m=+37.862235207" watchObservedRunningTime="2026-11-03 22:12:55.151641594 +0000 UTC m=+37.862705029"
Nov 03 22:12:55.804485 ip-10-0-139-73 kubenswrapper[2570]: I1103 22:12:55.804444    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:12:55.892398 ip-10-0-139-73 systemd[1]: Stopping Kubernetes Kubelet...
Nov 03 22:12:55.902596 ip-10-0-139-73 systemd[1]: kubelet.service: Deactivated successfully.
Nov 03 22:12:55.902760 ip-10-0-139-73 systemd[1]: Stopped Kubernetes Kubelet.
Nov 03 22:12:55.902808 ip-10-0-139-73 systemd[1]: kubelet.service: Consumed 2.331s CPU time.
Nov 03 22:12:55.917678 ip-10-0-139-73 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 22:12:56.005051 ip-10-0-139-73 kubenswrapper[4431]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 22:12:56.005051 ip-10-0-139-73 kubenswrapper[4431]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 22:12:56.005051 ip-10-0-139-73 kubenswrapper[4431]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 22:12:56.005051 ip-10-0-139-73 kubenswrapper[4431]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 22:12:56.007526 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007432    4431 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 22:12:56.007526 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007520    4431 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 22:12:56.007526 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007525    4431 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 22:12:56.007526 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007529    4431 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 22:12:56.007526 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007533    4431 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 22:12:56.007847 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007538    4431 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 22:12:56.007847 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007542    4431 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 22:12:56.007847 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007556    4431 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 22:12:56.007847 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007560    4431 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 22:12:56.007847 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007564    4431 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 22:12:56.007847 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007568    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 22:12:56.007847 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007572    4431 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 22:12:56.007847 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007577    4431 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 22:12:56.007847 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007580    4431 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 22:12:56.007847 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007584    4431 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 22:12:56.007847 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007588    4431 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 22:12:56.007847 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007592    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 22:12:56.007847 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007596    4431 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 22:12:56.007847 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007600    4431 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 22:12:56.007847 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007604    4431 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 22:12:56.007847 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007608    4431 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 22:12:56.007847 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007611    4431 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 22:12:56.007847 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007615    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 22:12:56.007847 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007619    4431 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 22:12:56.007847 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007623    4431 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 22:12:56.008739 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007627    4431 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 22:12:56.008739 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007631    4431 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 22:12:56.008739 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007634    4431 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 22:12:56.008739 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007638    4431 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 22:12:56.008739 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007642    4431 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 22:12:56.008739 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007646    4431 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 22:12:56.008739 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007650    4431 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 22:12:56.008739 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007654    4431 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 22:12:56.008739 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007658    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 22:12:56.008739 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007663    4431 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 22:12:56.008739 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007667    4431 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 22:12:56.008739 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007672    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 22:12:56.008739 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007676    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 22:12:56.008739 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007681    4431 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 22:12:56.008739 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007686    4431 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 22:12:56.008739 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007692    4431 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 22:12:56.008739 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007696    4431 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 22:12:56.008739 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007701    4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 22:12:56.008739 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007705    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 22:12:56.008739 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007709    4431 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 22:12:56.009634 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007713    4431 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 22:12:56.009634 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007718    4431 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 22:12:56.009634 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007722    4431 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 22:12:56.009634 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007727    4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 22:12:56.009634 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007732    4431 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 22:12:56.009634 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007736    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 22:12:56.009634 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007740    4431 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 22:12:56.009634 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007744    4431 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 22:12:56.009634 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007748    4431 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 22:12:56.009634 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007753    4431 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 22:12:56.009634 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007757    4431 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 22:12:56.009634 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007761    4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 22:12:56.009634 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007765    4431 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 22:12:56.009634 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007769    4431 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 22:12:56.009634 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007774    4431 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 22:12:56.009634 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007778    4431 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 22:12:56.009634 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007782    4431 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 22:12:56.009634 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007786    4431 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 22:12:56.009634 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007790    4431 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 22:12:56.009634 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007795    4431 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 22:12:56.010549 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007799    4431 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 22:12:56.010549 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007803    4431 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 22:12:56.010549 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007808    4431 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 22:12:56.010549 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007812    4431 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 22:12:56.010549 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007817    4431 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 22:12:56.010549 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007822    4431 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 22:12:56.010549 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007826    4431 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 22:12:56.010549 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007831    4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 22:12:56.010549 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007835    4431 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 22:12:56.010549 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007842    4431 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 22:12:56.010549 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007847    4431 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 22:12:56.010549 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007852    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 22:12:56.010549 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007857    4431 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 22:12:56.010549 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007861    4431 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 22:12:56.010549 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007867    4431 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 22:12:56.010549 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007871    4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 22:12:56.010549 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007876    4431 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 22:12:56.010549 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007880    4431 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 22:12:56.010549 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007885    4431 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 22:12:56.011436 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007889    4431 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 22:12:56.011436 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007893    4431 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 22:12:56.011436 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007898    4431 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 22:12:56.011436 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007902    4431 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 22:12:56.011436 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007906    4431 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 22:12:56.011436 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007912    4431 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 22:12:56.011436 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007916    4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 22:12:56.011436 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007921    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 22:12:56.011436 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007925    4431 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 22:12:56.011436 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007930    4431 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 22:12:56.011436 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007934    4431 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 22:12:56.011436 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007941    4431 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 22:12:56.011436 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007948    4431 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 22:12:56.011436 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007961    4431 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 22:12:56.011436 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007966    4431 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 22:12:56.011436 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007972    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 22:12:56.011436 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007977    4431 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 22:12:56.011436 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007983    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 22:12:56.011436 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007987    4431 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 22:12:56.012313 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007994    4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 22:12:56.012313 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.007998    4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 22:12:56.012313 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.008004    4431 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 22:12:56.012313 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009278    4431 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 22:12:56.012313 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009289    4431 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 22:12:56.012313 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009293    4431 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 22:12:56.012313 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009298    4431 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 22:12:56.012313 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009302    4431 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 22:12:56.012313 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009308    4431 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 22:12:56.012313 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009312    4431 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 22:12:56.012313 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009317    4431 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 22:12:56.012313 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009321    4431 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 22:12:56.012313 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009332    4431 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 22:12:56.012313 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009340    4431 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 22:12:56.012313 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009345    4431 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 22:12:56.012313 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009349    4431 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 22:12:56.012313 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009354    4431 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 22:12:56.012313 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009358    4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 22:12:56.012313 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009362    4431 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 22:12:56.013187 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009366    4431 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 22:12:56.013187 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009371    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 22:12:56.013187 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009375    4431 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 22:12:56.013187 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009379    4431 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 22:12:56.013187 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009384    4431 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 22:12:56.013187 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009388    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 22:12:56.013187 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009392    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 22:12:56.013187 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009396    4431 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 22:12:56.013187 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009400    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 22:12:56.013187 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009404    4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 22:12:56.013187 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009409    4431 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 22:12:56.013187 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009413    4431 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 22:12:56.013187 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009418    4431 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 22:12:56.013187 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009424    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 22:12:56.013187 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009428    4431 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 22:12:56.013187 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009432    4431 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 22:12:56.013187 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009438    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 22:12:56.013187 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009443    4431 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 22:12:56.013187 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009447    4431 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 22:12:56.014021 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009451    4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 22:12:56.014021 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009455    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 22:12:56.014021 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009459    4431 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 22:12:56.014021 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009464    4431 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 22:12:56.014021 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009468    4431 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 22:12:56.014021 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009473    4431 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 22:12:56.014021 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009479    4431 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 22:12:56.014021 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009484    4431 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 22:12:56.014021 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009488    4431 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 22:12:56.014021 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009492    4431 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 22:12:56.014021 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009496    4431 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 22:12:56.014021 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009500    4431 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 22:12:56.014021 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009505    4431 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 22:12:56.014021 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009510    4431 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 22:12:56.014021 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009514    4431 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 22:12:56.014021 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009518    4431 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 22:12:56.014021 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009522    4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 22:12:56.014021 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009527    4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 22:12:56.014021 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009531    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 22:12:56.014021 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009535    4431 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 22:12:56.014567 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009539    4431 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 22:12:56.014567 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009543    4431 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 22:12:56.014567 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009548    4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 22:12:56.014567 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009552    4431 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 22:12:56.014567 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009557    4431 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 22:12:56.014567 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009562    4431 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 22:12:56.014567 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009566    4431 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 22:12:56.014567 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009570    4431 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 22:12:56.014567 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009574    4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 22:12:56.014567 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009578    4431 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 22:12:56.014567 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009583    4431 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 22:12:56.014567 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009587    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 22:12:56.014567 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009591    4431 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 22:12:56.014567 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009596    4431 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 22:12:56.014567 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009600    4431 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 22:12:56.014567 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009604    4431 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 22:12:56.014567 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009609    4431 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 22:12:56.014567 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009615    4431 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 22:12:56.014567 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009621    4431 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 22:12:56.015040 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009625    4431 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 22:12:56.015040 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009629    4431 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 22:12:56.015040 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009633    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 22:12:56.015040 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009637    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 22:12:56.015040 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009642    4431 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 22:12:56.015040 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009646    4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 22:12:56.015040 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009650    4431 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 22:12:56.015040 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009654    4431 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 22:12:56.015040 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009659    4431 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 22:12:56.015040 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009663    4431 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 22:12:56.015040 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009668    4431 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 22:12:56.015040 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009672    4431 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 22:12:56.015040 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009676    4431 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 22:12:56.015040 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009680    4431 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 22:12:56.015040 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009684    4431 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 22:12:56.015040 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009691    4431 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 22:12:56.015040 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009697    4431 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 22:12:56.015040 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009701    4431 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 22:12:56.015040 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009706    4431 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 22:12:56.015560 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009710    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 22:12:56.015560 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009715    4431 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 22:12:56.015560 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009719    4431 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 22:12:56.015560 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009724    4431 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 22:12:56.015560 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009728    4431 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 22:12:56.015560 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009733    4431 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 22:12:56.015560 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009738    4431 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 22:12:56.015560 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009742    4431 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 22:12:56.015560 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009746    4431 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 22:12:56.015560 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009750    4431 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 22:12:56.015560 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009756    4431 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 22:12:56.015560 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009761    4431 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 22:12:56.015560 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.009765    4431 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 22:12:56.015560 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010596    4431 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 22:12:56.015560 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010605    4431 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 22:12:56.015560 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010610    4431 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 22:12:56.015560 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010614    4431 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 22:12:56.015560 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010619    4431 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 22:12:56.015560 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010623    4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 22:12:56.015560 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010628    4431 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 22:12:56.016117 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010632    4431 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 22:12:56.016117 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010636    4431 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 22:12:56.016117 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010641    4431 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 22:12:56.016117 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010645    4431 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 22:12:56.016117 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010649    4431 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 22:12:56.016117 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010653    4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 22:12:56.016117 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010657    4431 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 22:12:56.016117 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010663    4431 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 22:12:56.016117 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010669    4431 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 22:12:56.016117 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010674    4431 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 22:12:56.016117 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010680    4431 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 22:12:56.016117 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010684    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 22:12:56.016117 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010688    4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 22:12:56.016117 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010693    4431 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 22:12:56.016117 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010697    4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 22:12:56.016117 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010701    4431 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 22:12:56.016117 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010705    4431 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 22:12:56.016117 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010710    4431 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 22:12:56.016117 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010715    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 22:12:56.016117 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010719    4431 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 22:12:56.016877 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010724    4431 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 22:12:56.016877 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010728    4431 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 22:12:56.016877 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010732    4431 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 22:12:56.016877 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010737    4431 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 22:12:56.016877 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010741    4431 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 22:12:56.016877 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010746    4431 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 22:12:56.016877 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010750    4431 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 22:12:56.016877 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010754    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 22:12:56.016877 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010759    4431 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 22:12:56.016877 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010763    4431 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 22:12:56.016877 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010768    4431 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 22:12:56.016877 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010772    4431 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 22:12:56.016877 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010776    4431 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 22:12:56.016877 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010781    4431 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 22:12:56.016877 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010785    4431 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 22:12:56.016877 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010790    4431 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 22:12:56.016877 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010794    4431 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 22:12:56.016877 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010799    4431 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 22:12:56.016877 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010803    4431 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 22:12:56.017583 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010807    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 22:12:56.017583 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010812    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 22:12:56.017583 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010816    4431 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 22:12:56.017583 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010821    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 22:12:56.017583 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010826    4431 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 22:12:56.017583 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010830    4431 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 22:12:56.017583 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010835    4431 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 22:12:56.017583 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010841    4431 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 22:12:56.017583 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010848    4431 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 22:12:56.017583 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010854    4431 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 22:12:56.017583 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010860    4431 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 22:12:56.017583 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010866    4431 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 22:12:56.017583 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010871    4431 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 22:12:56.017583 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010876    4431 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 22:12:56.017583 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010880    4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 22:12:56.017583 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010884    4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 22:12:56.017583 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010888    4431 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 22:12:56.017583 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010894    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 22:12:56.017583 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010899    4431 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 22:12:56.018055 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010903    4431 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 22:12:56.018055 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010908    4431 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 22:12:56.018055 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010912    4431 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 22:12:56.018055 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010917    4431 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 22:12:56.018055 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010921    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 22:12:56.018055 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010925    4431 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 22:12:56.018055 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010929    4431 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 22:12:56.018055 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010933    4431 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 22:12:56.018055 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010937    4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 22:12:56.018055 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010942    4431 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 22:12:56.018055 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010946    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 22:12:56.018055 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010950    4431 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 22:12:56.018055 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010955    4431 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 22:12:56.018055 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010959    4431 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 22:12:56.018055 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010963    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 22:12:56.018055 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010967    4431 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 22:12:56.018055 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010974    4431 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 22:12:56.018055 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010978    4431 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 22:12:56.018055 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010982    4431 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 22:12:56.018055 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010986    4431 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 22:12:56.018573 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010991    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 22:12:56.018573 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010995    4431 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 22:12:56.018573 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.010999    4431 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 22:12:56.018573 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.011003    4431 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 22:12:56.018573 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.011007    4431 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 22:12:56.018573 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.011012    4431 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 22:12:56.018573 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.011017    4431 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 22:12:56.018573 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.011021    4431 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 22:12:56.018573 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.011025    4431 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 22:12:56.018573 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.011030    4431 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 22:12:56.018573 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.011034    4431 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 22:12:56.018573 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.011038    4431 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 22:12:56.018573 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.011042    4431 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 22:12:56.018573 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.011047    4431 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 22:12:56.018573 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.011051    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 22:12:56.018573 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.011055    4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 22:12:56.018573 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.011061    4431 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 22:12:56.018573 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.011065    4431 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 22:12:56.018573 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.011069    4431 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 22:12:56.018573 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.011074    4431 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 22:12:56.019065 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.011078    4431 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 22:12:56.019065 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011230    4431 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 22:12:56.019065 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011255    4431 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 22:12:56.019065 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011268    4431 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 22:12:56.019065 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011275    4431 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 22:12:56.019065 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011282    4431 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 22:12:56.019065 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011287    4431 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 22:12:56.019065 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011294    4431 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 22:12:56.019065 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011303    4431 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 22:12:56.019065 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011309    4431 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 22:12:56.019065 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011314    4431 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 22:12:56.019065 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011320    4431 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 22:12:56.019065 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011325    4431 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 22:12:56.019065 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011331    4431 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 22:12:56.019065 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011336    4431 flags.go:64] FLAG: --cgroup-root=""
Nov 03 22:12:56.019065 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011340    4431 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 22:12:56.019065 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011345    4431 flags.go:64] FLAG: --client-ca-file=""
Nov 03 22:12:56.019065 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011351    4431 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 22:12:56.019065 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011357    4431 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 22:12:56.019065 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011365    4431 flags.go:64] FLAG: --cluster-domain=""
Nov 03 22:12:56.019065 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011370    4431 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 22:12:56.019065 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011375    4431 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 22:12:56.019065 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011380    4431 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 22:12:56.019065 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011386    4431 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 22:12:56.019669 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011392    4431 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 22:12:56.019669 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011397    4431 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 22:12:56.019669 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011402    4431 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 22:12:56.019669 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011408    4431 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 22:12:56.019669 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011413    4431 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 22:12:56.019669 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011418    4431 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 22:12:56.019669 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011423    4431 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 22:12:56.019669 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011428    4431 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 22:12:56.019669 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011433    4431 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 22:12:56.019669 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011440    4431 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 22:12:56.019669 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011445    4431 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 22:12:56.019669 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011451    4431 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 22:12:56.019669 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011456    4431 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 22:12:56.019669 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011460    4431 flags.go:64] FLAG: --enable-server="true"
Nov 03 22:12:56.019669 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011465    4431 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 22:12:56.019669 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011472    4431 flags.go:64] FLAG: --event-burst="100"
Nov 03 22:12:56.019669 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011479    4431 flags.go:64] FLAG: --event-qps="50"
Nov 03 22:12:56.019669 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011484    4431 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 22:12:56.019669 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011489    4431 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 22:12:56.019669 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011494    4431 flags.go:64] FLAG: --eviction-hard=""
Nov 03 22:12:56.019669 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011500    4431 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 22:12:56.019669 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011505    4431 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 22:12:56.019669 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011510    4431 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 22:12:56.019669 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011515    4431 flags.go:64] FLAG: --eviction-soft=""
Nov 03 22:12:56.019669 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011520    4431 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 22:12:56.020306 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011525    4431 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 22:12:56.020306 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011530    4431 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 22:12:56.020306 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011536    4431 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 22:12:56.020306 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011541    4431 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 22:12:56.020306 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011546    4431 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 22:12:56.020306 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011551    4431 flags.go:64] FLAG: --feature-gates=""
Nov 03 22:12:56.020306 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011558    4431 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 22:12:56.020306 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011563    4431 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 22:12:56.020306 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011569    4431 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 22:12:56.020306 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011574    4431 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 22:12:56.020306 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011579    4431 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 22:12:56.020306 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011584    4431 flags.go:64] FLAG: --help="false"
Nov 03 22:12:56.020306 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011589    4431 flags.go:64] FLAG: --hostname-override="ip-10-0-139-73.ec2.internal"
Nov 03 22:12:56.020306 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011594    4431 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 22:12:56.020306 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011599    4431 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 22:12:56.020306 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011604    4431 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 22:12:56.020306 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011610    4431 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 22:12:56.020306 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011616    4431 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 22:12:56.020306 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011622    4431 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 22:12:56.020306 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011627    4431 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 22:12:56.020306 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011631    4431 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 22:12:56.020306 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011636    4431 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 22:12:56.020306 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011641    4431 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 22:12:56.020858 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011652    4431 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 22:12:56.020858 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011657    4431 flags.go:64] FLAG: --kube-reserved=""
Nov 03 22:12:56.020858 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011662    4431 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 22:12:56.020858 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011666    4431 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 22:12:56.020858 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011671    4431 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 22:12:56.020858 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011676    4431 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 22:12:56.020858 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011681    4431 flags.go:64] FLAG: --lock-file=""
Nov 03 22:12:56.020858 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011686    4431 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 22:12:56.020858 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011691    4431 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 22:12:56.020858 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011696    4431 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 22:12:56.020858 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011712    4431 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 22:12:56.020858 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011717    4431 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 22:12:56.020858 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011725    4431 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 22:12:56.020858 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011730    4431 flags.go:64] FLAG: --logging-format="text"
Nov 03 22:12:56.020858 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011735    4431 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 22:12:56.020858 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011741    4431 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 22:12:56.020858 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011745    4431 flags.go:64] FLAG: --manifest-url=""
Nov 03 22:12:56.020858 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011750    4431 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 22:12:56.020858 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011757    4431 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 22:12:56.020858 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011762    4431 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 22:12:56.020858 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011769    4431 flags.go:64] FLAG: --max-pods="110"
Nov 03 22:12:56.020858 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011774    4431 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 22:12:56.020858 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011780    4431 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 22:12:56.020858 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011784    4431 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 22:12:56.020858 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011789    4431 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 22:12:56.021533 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011795    4431 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 22:12:56.021533 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011800    4431 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 22:12:56.021533 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011806    4431 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 22:12:56.021533 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011817    4431 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 22:12:56.021533 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011823    4431 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 22:12:56.021533 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011828    4431 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 22:12:56.021533 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011833    4431 flags.go:64] FLAG: --pod-cidr=""
Nov 03 22:12:56.021533 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011841    4431 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 22:12:56.021533 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011846    4431 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 22:12:56.021533 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011851    4431 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 22:12:56.021533 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011855    4431 flags.go:64] FLAG: --port="10250"
Nov 03 22:12:56.021533 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011861    4431 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 22:12:56.021533 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011866    4431 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-09a331f3e94331be7"
Nov 03 22:12:56.021533 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011871    4431 flags.go:64] FLAG: --qos-reserved=""
Nov 03 22:12:56.021533 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011876    4431 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 22:12:56.021533 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011882    4431 flags.go:64] FLAG: --register-node="true"
Nov 03 22:12:56.021533 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011887    4431 flags.go:64] FLAG: --register-with-taints=""
Nov 03 22:12:56.021533 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011893    4431 flags.go:64] FLAG: --registry-burst="10"
Nov 03 22:12:56.021533 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011897    4431 flags.go:64] FLAG: --registry-qps="5"
Nov 03 22:12:56.021533 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011902    4431 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 22:12:56.021533 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011908    4431 flags.go:64] FLAG: --reserved-memory=""
Nov 03 22:12:56.021533 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011914    4431 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 22:12:56.021533 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011919    4431 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 22:12:56.021533 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011924    4431 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 22:12:56.021533 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011929    4431 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 22:12:56.021533 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011934    4431 flags.go:64] FLAG: --runonce="false"
Nov 03 22:12:56.022307 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011939    4431 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 22:12:56.022307 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011944    4431 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 22:12:56.022307 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011948    4431 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 22:12:56.022307 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011960    4431 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 22:12:56.022307 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011965    4431 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 22:12:56.022307 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011971    4431 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 22:12:56.022307 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011976    4431 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 22:12:56.022307 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011981    4431 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 22:12:56.022307 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011988    4431 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 22:12:56.022307 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011992    4431 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 22:12:56.022307 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.011997    4431 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 22:12:56.022307 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.012002    4431 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 22:12:56.022307 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.012007    4431 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 22:12:56.022307 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.012014    4431 flags.go:64] FLAG: --system-cgroups=""
Nov 03 22:12:56.022307 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.012019    4431 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 22:12:56.022307 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.012027    4431 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 22:12:56.022307 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.012032    4431 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 22:12:56.022307 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.012051    4431 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 22:12:56.022307 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.012059    4431 flags.go:64] FLAG: --tls-min-version=""
Nov 03 22:12:56.022307 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.012064    4431 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 22:12:56.022307 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.012068    4431 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 22:12:56.022307 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.012073    4431 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 22:12:56.022307 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.012078    4431 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 22:12:56.022307 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.012083    4431 flags.go:64] FLAG: --v="2"
Nov 03 22:12:56.022307 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.012090    4431 flags.go:64] FLAG: --version="false"
Nov 03 22:12:56.022921 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.012097    4431 flags.go:64] FLAG: --vmodule=""
Nov 03 22:12:56.022921 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.012104    4431 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 22:12:56.022921 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.012110    4431 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 22:12:56.022921 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012367    4431 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 22:12:56.022921 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012375    4431 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 22:12:56.022921 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012380    4431 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 22:12:56.022921 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012384    4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 22:12:56.022921 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012389    4431 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 22:12:56.022921 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012394    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 22:12:56.022921 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012398    4431 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 22:12:56.022921 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012403    4431 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 22:12:56.022921 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012407    4431 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 22:12:56.022921 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012411    4431 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 22:12:56.022921 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012416    4431 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 22:12:56.022921 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012420    4431 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 22:12:56.022921 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012427    4431 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 22:12:56.022921 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012431    4431 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 22:12:56.022921 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012436    4431 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 22:12:56.022921 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012440    4431 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 22:12:56.022921 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012445    4431 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 22:12:56.023463 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012451    4431 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 22:12:56.023463 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012456    4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 22:12:56.023463 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012460    4431 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 22:12:56.023463 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012464    4431 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 22:12:56.023463 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012468    4431 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 22:12:56.023463 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012473    4431 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 22:12:56.023463 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012477    4431 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 22:12:56.023463 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012482    4431 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 22:12:56.023463 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012487    4431 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 22:12:56.023463 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012491    4431 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 22:12:56.023463 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012496    4431 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 22:12:56.023463 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012500    4431 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 22:12:56.023463 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012505    4431 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 22:12:56.023463 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012510    4431 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 22:12:56.023463 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012514    4431 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 22:12:56.023463 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012520    4431 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 22:12:56.023463 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012525    4431 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 22:12:56.023463 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012529    4431 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 22:12:56.023463 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012533    4431 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 22:12:56.023463 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012538    4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 22:12:56.024001 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012542    4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 22:12:56.024001 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012547    4431 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 22:12:56.024001 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012551    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 22:12:56.024001 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012555    4431 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 22:12:56.024001 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012559    4431 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 22:12:56.024001 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012564    4431 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 22:12:56.024001 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012568    4431 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 22:12:56.024001 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012575    4431 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 22:12:56.024001 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012580    4431 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 22:12:56.024001 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012584    4431 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 22:12:56.024001 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012588    4431 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 22:12:56.024001 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012593    4431 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 22:12:56.024001 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012599    4431 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 22:12:56.024001 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012603    4431 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 22:12:56.024001 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012607    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 22:12:56.024001 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012612    4431 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 22:12:56.024001 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012616    4431 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 22:12:56.024001 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012623    4431 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 22:12:56.024001 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012628    4431 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 22:12:56.024001 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012633    4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 22:12:56.024604 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012638    4431 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 22:12:56.024604 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012642    4431 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 22:12:56.024604 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012646    4431 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 22:12:56.024604 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012651    4431 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 22:12:56.024604 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012656    4431 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 22:12:56.024604 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012660    4431 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 22:12:56.024604 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012665    4431 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 22:12:56.024604 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012669    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 22:12:56.024604 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012674    4431 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 22:12:56.024604 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012679    4431 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 22:12:56.024604 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012683    4431 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 22:12:56.024604 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012687    4431 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 22:12:56.024604 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012691    4431 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 22:12:56.024604 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012695    4431 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 22:12:56.024604 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012700    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 22:12:56.024604 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012705    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 22:12:56.024604 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012709    4431 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 22:12:56.024604 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012713    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 22:12:56.024604 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012717    4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 22:12:56.024604 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012723    4431 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 22:12:56.025139 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012728    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 22:12:56.025139 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012732    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 22:12:56.025139 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012736    4431 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 22:12:56.025139 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012740    4431 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 22:12:56.025139 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012746    4431 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 22:12:56.025139 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012751    4431 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 22:12:56.025139 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012755    4431 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 22:12:56.025139 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012760    4431 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 22:12:56.025139 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012764    4431 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 22:12:56.025139 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012768    4431 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 22:12:56.025139 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012773    4431 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 22:12:56.025139 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012777    4431 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 22:12:56.025139 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012781    4431 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 22:12:56.025139 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012786    4431 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 22:12:56.025139 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012790    4431 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 22:12:56.025139 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012794    4431 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 22:12:56.025139 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012799    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 22:12:56.025139 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012803    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 22:12:56.025139 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012807    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 22:12:56.025643 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012811    4431 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 22:12:56.025643 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012816    4431 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 22:12:56.025643 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012821    4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 22:12:56.025643 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012825    4431 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 22:12:56.025643 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012829    4431 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 22:12:56.025643 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012834    4431 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 22:12:56.025643 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012840    4431 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 22:12:56.025643 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012846    4431 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 22:12:56.025643 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012851    4431 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 22:12:56.025643 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012855    4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 22:12:56.025643 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.012898    4431 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 22:12:56.025643 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012960    4431 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 22:12:56.025643 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012968    4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 22:12:56.025643 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012974    4431 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 22:12:56.025643 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012979    4431 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 22:12:56.026120 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012983    4431 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 22:12:56.026120 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012988    4431 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 22:12:56.026120 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012994    4431 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 22:12:56.026120 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.012998    4431 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 22:12:56.026120 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013003    4431 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 22:12:56.026120 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013007    4431 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 22:12:56.026120 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013012    4431 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 22:12:56.026120 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013016    4431 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 22:12:56.026120 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013021    4431 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 22:12:56.026120 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013026    4431 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 22:12:56.026120 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013030    4431 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 22:12:56.026120 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013035    4431 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 22:12:56.026120 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013039    4431 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 22:12:56.026120 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013043    4431 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 22:12:56.026120 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013048    4431 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 22:12:56.026120 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013052    4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 22:12:56.026120 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013057    4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 22:12:56.026120 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013062    4431 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 22:12:56.026120 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013066    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 22:12:56.026120 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013070    4431 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 22:12:56.026718 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013075    4431 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 22:12:56.026718 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013080    4431 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 22:12:56.026718 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013084    4431 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 22:12:56.026718 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013089    4431 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 22:12:56.026718 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013093    4431 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 22:12:56.026718 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013097    4431 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 22:12:56.026718 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013102    4431 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 22:12:56.026718 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013106    4431 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 22:12:56.026718 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013111    4431 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 22:12:56.026718 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013116    4431 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 22:12:56.026718 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013122    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 22:12:56.026718 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013126    4431 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 22:12:56.026718 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013131    4431 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 22:12:56.026718 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013137    4431 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 22:12:56.026718 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013145    4431 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 22:12:56.026718 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013149    4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 22:12:56.026718 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013154    4431 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 22:12:56.026718 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013158    4431 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 22:12:56.026718 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013163    4431 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 22:12:56.026718 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013167    4431 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 22:12:56.027274 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013171    4431 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 22:12:56.027274 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013176    4431 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 22:12:56.027274 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013180    4431 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 22:12:56.027274 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013184    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 22:12:56.027274 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013189    4431 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 22:12:56.027274 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013193    4431 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 22:12:56.027274 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013197    4431 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 22:12:56.027274 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013201    4431 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 22:12:56.027274 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013205    4431 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 22:12:56.027274 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013210    4431 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 22:12:56.027274 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013232    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 22:12:56.027274 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013236    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 22:12:56.027274 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013241    4431 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 22:12:56.027274 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013246    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 22:12:56.027274 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013251    4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 22:12:56.027274 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013255    4431 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 22:12:56.027274 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013259    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 22:12:56.027274 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013263    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 22:12:56.027274 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013268    4431 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 22:12:56.027739 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013272    4431 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 22:12:56.027739 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013276    4431 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 22:12:56.027739 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013280    4431 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 22:12:56.027739 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013287    4431 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 22:12:56.027739 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013291    4431 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 22:12:56.027739 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013296    4431 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 22:12:56.027739 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013300    4431 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 22:12:56.027739 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013306    4431 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 22:12:56.027739 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013310    4431 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 22:12:56.027739 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013315    4431 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 22:12:56.027739 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013319    4431 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 22:12:56.027739 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013323    4431 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 22:12:56.027739 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013328    4431 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 22:12:56.027739 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013333    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 22:12:56.027739 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013337    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 22:12:56.027739 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013341    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 22:12:56.027739 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013345    4431 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 22:12:56.027739 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013350    4431 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 22:12:56.027739 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013354    4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 22:12:56.028235 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013358    4431 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 22:12:56.028235 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013362    4431 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 22:12:56.028235 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013367    4431 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 22:12:56.028235 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013375    4431 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 22:12:56.028235 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013382    4431 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 22:12:56.028235 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013388    4431 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 22:12:56.028235 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013393    4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 22:12:56.028235 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013398    4431 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 22:12:56.028235 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013403    4431 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 22:12:56.028235 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013408    4431 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 22:12:56.028235 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013412    4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 22:12:56.028235 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013416    4431 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 22:12:56.028235 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013420    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 22:12:56.028235 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013425    4431 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 22:12:56.028235 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013429    4431 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 22:12:56.028235 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013433    4431 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 22:12:56.028235 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013439    4431 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 22:12:56.028235 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013443    4431 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 22:12:56.028235 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013448    4431 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 22:12:56.028704 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013452    4431 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 22:12:56.028704 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013458    4431 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 22:12:56.028704 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013463    4431 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 22:12:56.028704 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013467    4431 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 22:12:56.028704 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.013472    4431 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 22:12:56.028704 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.018674    4431 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 22:12:56.028704 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.018693    4431 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 22:12:56.028704 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.018712    4431 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 22:12:56.028704 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.018718    4431 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 22:12:56.028704 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.018778    4431 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 22:12:56.028704 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.018783    4431 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 22:12:56.028704 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.018786    4431 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 22:12:56.028704 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.018790    4431 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 22:12:56.028704 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.018793    4431 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 22:12:56.028704 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.018796    4431 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 22:12:56.028704 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.018798    4431 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 22:12:56.028704 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.018801    4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 22:12:56.028704 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.018804    4431 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 22:12:56.028704 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.018807    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 22:12:56.029169 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.018809    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 22:12:56.029169 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.018812    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 22:12:56.029169 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.018815    4431 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 22:12:56.029169 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.018818    4431 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 22:12:56.029169 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.018821    4431 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 22:12:56.029169 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.018824    4431 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 22:12:56.029169 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.018827    4431 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 22:12:56.029169 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.018829    4431 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 22:12:56.029169 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.018833    4431 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 22:12:56.029169 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.018835    4431 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 22:12:56.029169 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.018838    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 22:12:56.029169 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.018841    4431 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 22:12:56.029169 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.018843    4431 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 22:12:56.029169 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.018846    4431 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 22:12:56.029169 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.018849    4431 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 22:12:56.029169 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.018852    4431 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 22:12:56.029169 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.018855    4431 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 22:12:56.029169 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.018858    4431 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 22:12:56.029169 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.018860    4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 22:12:56.029169 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.018863    4431 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 22:12:56.029778 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.018866    4431 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 22:12:56.029778 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.018869    4431 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 22:12:56.029778 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.018872    4431 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 22:12:56.029778 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.018875    4431 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 22:12:56.029778 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.018878    4431 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 22:12:56.029778 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.018880    4431 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 22:12:56.029778 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.018883    4431 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 22:12:56.029778 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.018885    4431 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 22:12:56.029778 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.018888    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 22:12:56.029778 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.018891    4431 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 22:12:56.029778 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.018894    4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 22:12:56.029778 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.018896    4431 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 22:12:56.029778 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.018899    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 22:12:56.029778 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.018901    4431 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 22:12:56.029778 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.018904    4431 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 22:12:56.029778 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.018907    4431 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 22:12:56.029778 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.018910    4431 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 22:12:56.029778 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.018913    4431 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 22:12:56.029778 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.018915    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 22:12:56.030254 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.018918    4431 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 22:12:56.030254 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.018921    4431 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 22:12:56.030254 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.018923    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 22:12:56.030254 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.018926    4431 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 22:12:56.030254 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.018928    4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 22:12:56.030254 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.018931    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 22:12:56.030254 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.018933    4431 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 22:12:56.030254 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.018936    4431 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 22:12:56.030254 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.018939    4431 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 22:12:56.030254 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.018942    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 22:12:56.030254 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.018945    4431 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 22:12:56.030254 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.018947    4431 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 22:12:56.030254 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.018950    4431 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 22:12:56.030254 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.018952    4431 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 22:12:56.030254 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.018955    4431 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 22:12:56.030254 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.018958    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 22:12:56.030254 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.018961    4431 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 22:12:56.030254 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.018963    4431 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 22:12:56.030254 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.018966    4431 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 22:12:56.030254 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.018968    4431 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 22:12:56.030797 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.018971    4431 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 22:12:56.030797 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.018975    4431 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 22:12:56.030797 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.018979    4431 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 22:12:56.030797 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.018983    4431 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 22:12:56.030797 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.018986    4431 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 22:12:56.030797 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.018989    4431 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 22:12:56.030797 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.018992    4431 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 22:12:56.030797 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.018994    4431 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 22:12:56.030797 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.018997    4431 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 22:12:56.030797 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019000    4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 22:12:56.030797 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019004    4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 22:12:56.030797 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019006    4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 22:12:56.030797 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019009    4431 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 22:12:56.030797 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019012    4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 22:12:56.030797 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019015    4431 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 22:12:56.030797 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019017    4431 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 22:12:56.030797 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019020    4431 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 22:12:56.030797 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019023    4431 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 22:12:56.030797 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019026    4431 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 22:12:56.031276 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019029    4431 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 22:12:56.031276 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019031    4431 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 22:12:56.031276 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019035    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 22:12:56.031276 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019037    4431 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 22:12:56.031276 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019040    4431 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 22:12:56.031276 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019042    4431 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 22:12:56.031276 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019045    4431 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 22:12:56.031276 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019048    4431 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 22:12:56.031276 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019052    4431 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 22:12:56.031276 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019055    4431 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 22:12:56.031276 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019058    4431 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 22:12:56.031276 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019060    4431 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 22:12:56.031276 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019063    4431 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 22:12:56.031276 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019065    4431 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 22:12:56.031276 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019068    4431 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 22:12:56.031276 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019071    4431 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 22:12:56.031276 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019073    4431 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 22:12:56.031276 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019076    4431 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 22:12:56.031728 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.019105    4431 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 22:12:56.031728 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019290    4431 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 22:12:56.031728 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019296    4431 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 22:12:56.031728 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019299    4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 22:12:56.031728 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019302    4431 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 22:12:56.031728 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019305    4431 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 22:12:56.031728 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019309    4431 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 22:12:56.031728 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019311    4431 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 22:12:56.031728 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019314    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 22:12:56.031728 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019317    4431 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 22:12:56.031728 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019319    4431 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 22:12:56.031728 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019322    4431 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 22:12:56.031728 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019324    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 22:12:56.031728 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019327    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 22:12:56.031728 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019330    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 22:12:56.031728 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019333    4431 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 22:12:56.032236 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019336    4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 22:12:56.032236 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019339    4431 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 22:12:56.032236 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019342    4431 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 22:12:56.032236 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019345    4431 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 22:12:56.032236 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019347    4431 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 22:12:56.032236 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019350    4431 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 22:12:56.032236 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019353    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 22:12:56.032236 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019355    4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 22:12:56.032236 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019358    4431 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 22:12:56.032236 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019360    4431 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 22:12:56.032236 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019363    4431 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 22:12:56.032236 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019366    4431 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 22:12:56.032236 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019368    4431 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 22:12:56.032236 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019371    4431 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 22:12:56.032236 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019373    4431 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 22:12:56.032236 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019376    4431 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 22:12:56.032236 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019379    4431 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 22:12:56.032236 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019381    4431 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 22:12:56.032236 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019384    4431 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 22:12:56.032236 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019386    4431 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 22:12:56.033007 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019389    4431 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 22:12:56.033007 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019391    4431 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 22:12:56.033007 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019394    4431 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 22:12:56.033007 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019397    4431 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 22:12:56.033007 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019400    4431 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 22:12:56.033007 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019403    4431 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 22:12:56.033007 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019405    4431 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 22:12:56.033007 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019409    4431 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 22:12:56.033007 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019414    4431 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 22:12:56.033007 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019417    4431 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 22:12:56.033007 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019419    4431 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 22:12:56.033007 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019422    4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 22:12:56.033007 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019425    4431 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 22:12:56.033007 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019428    4431 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 22:12:56.033007 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019430    4431 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 22:12:56.033007 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019433    4431 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 22:12:56.033007 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019435    4431 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 22:12:56.033007 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019438    4431 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 22:12:56.033007 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019440    4431 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 22:12:56.033007 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019442    4431 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 22:12:56.033543 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019445    4431 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 22:12:56.033543 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019447    4431 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 22:12:56.033543 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019450    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 22:12:56.033543 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019453    4431 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 22:12:56.033543 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019455    4431 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 22:12:56.033543 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019458    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 22:12:56.033543 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019460    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 22:12:56.033543 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019463    4431 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 22:12:56.033543 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019466    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 22:12:56.033543 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019468    4431 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 22:12:56.033543 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019471    4431 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 22:12:56.033543 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019473    4431 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 22:12:56.033543 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019476    4431 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 22:12:56.033543 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019478    4431 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 22:12:56.033543 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019481    4431 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 22:12:56.033543 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019484    4431 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 22:12:56.033543 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019487    4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 22:12:56.033543 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019490    4431 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 22:12:56.033543 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019492    4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 22:12:56.034001 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019495    4431 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 22:12:56.034001 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019498    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 22:12:56.034001 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019500    4431 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 22:12:56.034001 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019502    4431 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 22:12:56.034001 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019505    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 22:12:56.034001 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019508    4431 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 22:12:56.034001 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019510    4431 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 22:12:56.034001 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019513    4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 22:12:56.034001 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019516    4431 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 22:12:56.034001 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019519    4431 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 22:12:56.034001 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019521    4431 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 22:12:56.034001 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019524    4431 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 22:12:56.034001 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019526    4431 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 22:12:56.034001 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019529    4431 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 22:12:56.034001 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019531    4431 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 22:12:56.034001 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019534    4431 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 22:12:56.034001 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019536    4431 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 22:12:56.034001 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019539    4431 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 22:12:56.034001 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019541    4431 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 22:12:56.034637 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019544    4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 22:12:56.034637 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019546    4431 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 22:12:56.034637 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019549    4431 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 22:12:56.034637 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019552    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 22:12:56.034637 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019555    4431 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 22:12:56.034637 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019557    4431 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 22:12:56.034637 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019561    4431 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 22:12:56.034637 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019565    4431 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 22:12:56.034637 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019568    4431 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 22:12:56.034637 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019571    4431 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 22:12:56.034637 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019574    4431 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 22:12:56.034637 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019577    4431 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 22:12:56.034637 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019580    4431 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 22:12:56.034637 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.019606    4431 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 22:12:56.034637 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019652    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 22:12:56.034999 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019656    4431 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 22:12:56.034999 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019658    4431 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 22:12:56.034999 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019661    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 22:12:56.034999 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019664    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 22:12:56.034999 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019667    4431 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 22:12:56.034999 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019669    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 22:12:56.034999 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019672    4431 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 22:12:56.034999 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019675    4431 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 22:12:56.034999 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019678    4431 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 22:12:56.034999 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019681    4431 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 22:12:56.034999 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019683    4431 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 22:12:56.034999 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019686    4431 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 22:12:56.034999 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019689    4431 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 22:12:56.034999 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019692    4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 22:12:56.034999 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019694    4431 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 22:12:56.034999 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019697    4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 22:12:56.034999 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019699    4431 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 22:12:56.034999 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019702    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 22:12:56.034999 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019704    4431 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 22:12:56.035485 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019707    4431 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 22:12:56.035485 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019710    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 22:12:56.035485 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019712    4431 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 22:12:56.035485 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019715    4431 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 22:12:56.035485 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019718    4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 22:12:56.035485 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019720    4431 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 22:12:56.035485 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019723    4431 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 22:12:56.035485 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019725    4431 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 22:12:56.035485 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019728    4431 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 22:12:56.035485 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019731    4431 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 22:12:56.035485 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019734    4431 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 22:12:56.035485 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019736    4431 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 22:12:56.035485 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019739    4431 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 22:12:56.035485 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019742    4431 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 22:12:56.035485 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019744    4431 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 22:12:56.035485 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019747    4431 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 22:12:56.035485 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019749    4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 22:12:56.035485 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019752    4431 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 22:12:56.035485 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019754    4431 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 22:12:56.035959 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019757    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 22:12:56.035959 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019762    4431 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 22:12:56.035959 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019765    4431 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 22:12:56.035959 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019769    4431 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 22:12:56.035959 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019773    4431 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 22:12:56.035959 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019776    4431 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 22:12:56.035959 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019779    4431 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 22:12:56.035959 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019781    4431 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 22:12:56.035959 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019784    4431 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 22:12:56.035959 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019787    4431 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 22:12:56.035959 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019789    4431 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 22:12:56.035959 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019792    4431 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 22:12:56.035959 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019795    4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 22:12:56.035959 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019797    4431 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 22:12:56.035959 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019800    4431 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 22:12:56.035959 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019802    4431 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 22:12:56.035959 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019805    4431 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 22:12:56.035959 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019808    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 22:12:56.035959 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019811    4431 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 22:12:56.035959 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019813    4431 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 22:12:56.036556 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019816    4431 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 22:12:56.036556 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019818    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 22:12:56.036556 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019821    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 22:12:56.036556 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019823    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 22:12:56.036556 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019827    4431 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 22:12:56.036556 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019830    4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 22:12:56.036556 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019832    4431 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 22:12:56.036556 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019835    4431 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 22:12:56.036556 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019839    4431 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 22:12:56.036556 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019842    4431 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 22:12:56.036556 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019844    4431 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 22:12:56.036556 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019847    4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 22:12:56.036556 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019849    4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 22:12:56.036556 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019853    4431 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 22:12:56.036556 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019856    4431 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 22:12:56.036556 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019859    4431 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 22:12:56.036556 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019861    4431 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 22:12:56.036556 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019864    4431 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 22:12:56.036556 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019867    4431 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 22:12:56.036556 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019869    4431 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 22:12:56.037162 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019872    4431 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 22:12:56.037162 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019875    4431 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 22:12:56.037162 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019877    4431 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 22:12:56.037162 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019880    4431 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 22:12:56.037162 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019882    4431 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 22:12:56.037162 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019885    4431 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 22:12:56.037162 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019888    4431 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 22:12:56.037162 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019890    4431 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 22:12:56.037162 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019893    4431 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 22:12:56.037162 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019896    4431 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 22:12:56.037162 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019899    4431 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 22:12:56.037162 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019901    4431 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 22:12:56.037162 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019906    4431 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 22:12:56.037162 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019909    4431 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 22:12:56.037162 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019912    4431 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 22:12:56.037162 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019914    4431 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 22:12:56.037162 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019917    4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 22:12:56.037162 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019920    4431 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 22:12:56.037162 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019923    4431 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 22:12:56.037162 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019925    4431 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 22:12:56.037869 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019929    4431 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 22:12:56.037869 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019931    4431 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 22:12:56.037869 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019934    4431 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 22:12:56.037869 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019937    4431 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 22:12:56.037869 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019939    4431 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 22:12:56.037869 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019943    4431 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 22:12:56.037869 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:56.019945    4431 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 22:12:56.037869 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.020012    4431 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 22:12:56.037869 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.020984    4431 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 03 22:12:56.037869 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.021043    4431 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 03 22:12:56.037869 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.021230    4431 server.go:1014] "Starting client certificate rotation"
Nov 03 22:12:56.037869 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.021323    4431 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 22:12:56.037869 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.021372    4431 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 22:07:17 +0000 UTC" deadline="2028-04-18 17:45:17.912556259 +0000 UTC"
Nov 03 22:12:56.037869 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.021408    4431 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12763h32m21.89115101s"
Nov 03 22:12:56.037869 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.021765    4431 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 22:12:56.037869 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.022825    4431 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 22:12:56.038283 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.024236    4431 log.go:25] "Validated CRI v1 runtime API"
Nov 03 22:12:56.038283 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.025748    4431 log.go:25] "Validated CRI v1 image API"
Nov 03 22:12:56.038283 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.026571    4431 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 22:12:56.038283 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.028051    4431 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 e501a563-9a27-497f-9c02-7f2fa7b068d2:/dev/nvme0n1p3 ecdc7f09-2dd5-497e-9bdc-adf35550ec88:/dev/nvme0n1p4]
Nov 03 22:12:56.038397 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.028070    4431 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/01527eb9e50404bb5f91b695b61e185b6e7cf0927fd0d1252a2bb9d0c36ddc0e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/01527eb9e50404bb5f91b695b61e185b6e7cf0927fd0d1252a2bb9d0c36ddc0e/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0452dadfd2655386a337e6ef379ca20341938653d5f53aa2545e56566fe7e56a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0452dadfd2655386a337e6ef379ca20341938653d5f53aa2545e56566fe7e56a/userdata/shm major:0 minor:218 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/05e93303e59fe467c6f7d583564effdd8a09180902c8db08e8211ba9ad1b3861/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/05e93303e59fe467c6f7d583564effdd8a09180902c8db08e8211ba9ad1b3861/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/382c0ab10609a37b4a1f8e32d5dac3760c552c63c3a60682227fb3bfdaafca3c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/382c0ab10609a37b4a1f8e32d5dac3760c552c63c3a60682227fb3bfdaafca3c/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/46377aeffc4cd3d572a6c7b722747c8c24bbebb5583a67d3a04edfb6df236e07/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/46377aeffc4cd3d572a6c7b722747c8c24bbebb5583a67d3a04edfb6df236e07/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/537a5c4de9dae0664356e0290f054d99f3eefc22cc0b7682faa7cd0146fa2da5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/537a5c4de9dae0664356e0290f054d99f3eefc22cc0b7682faa7cd0146fa2da5/userdata/shm major:0 minor:210 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5a9c9ef3d251d426e883ac5835699c7f59588d1b12d6f68ced725e4607dbdaf5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5a9c9ef3d251d426e883ac5835699c7f59588d1b12d6f68ced725e4607dbdaf5/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/69a840eb07f498030ab8307ebdd6e0e3f3dd7188385aa3f7dec3998e7b8b3c21/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/69a840eb07f498030ab8307ebdd6e0e3f3dd7188385aa3f7dec3998e7b8b3c21/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/73a63c22ffae1c6c4abf49a45190081732a2518699b857e09e444501a9ed590a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/73a63c22ffae1c6c4abf49a45190081732a2518699b857e09e444501a9ed590a/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7b9044ebe11612a0e9eded07aab7df7c563a18a145b86286b40ed006dfb27949/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7b9044ebe11612a0e9eded07aab7df7c563a18a145b86286b40ed006dfb27949/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9427fc09e99178a5f76492a05fc10f074027d4022ad640cc3a535c624220f6d8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9427fc09e99178a5f76492a05fc10f074027d4022ad640cc3a535c624220f6d8/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/972ef7c557baa631f2303101cdfb55622fd8b8ceea4288d2f527e38c4bd4dfa8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/972ef7c557baa631f2303101cdfb55622fd8b8ceea4288d2f527e38c4bd4dfa8/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cbd1adc5425c7bbbc9dbc85c8ea9f67b3383c6657e7b8dc4ed932383257eed4a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cbd1adc5425c7bbbc9dbc85c8ea9f67b3383c6657e7b8dc4ed932383257eed4a/userdata/shm major:0 minor:222 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d02c882efc6b58702c439be61f776a0b522e52d436f895cce65cb3ff7d585d6b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d02c882efc6b58702c439be61f776a0b522e52d436f895cce65cb3ff7d585d6b/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/089f9a4b-8eec-4758-811b-e506b41cecbb/volumes/kubernetes.io~projected/kube-api-access-7bjhs:{mountpoint:/var/lib/kubelet/pods/089f9a4b-8eec-4758-811b-e506b41cecbb/volumes/kubernetes.io~projected/kube-api-access-7bjhs major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1b792dc0-5e1c-40c4-a6c6-f68b4c6c8aac/volumes/kubernetes.io~projected/kube-api-access-qdtpv:{mountpoint:/var/lib/kubelet/pods/1b792dc0-5e1c-40c4-a6c6-f68b4c6c8aac/volumes/kubernetes.io~projected/kube-api-access-qdtpv major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/309c2503-3fef-4048-af01-6b3b70aa52a1/volumes/kubernetes.io~projected/kube-api-access-8slxn:{mountpoint:/var/lib/kubelet/pods/309c2503-3fef-4048-af01-6b3b70aa52a1/volumes/kubernetes.io~projected/kube-api-access-8slxn major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/365a1938-752c-4cec-a917-c5a46c719307/volumes/kubernetes.io~projected/kube-api-access-qdv4k:{mountpoint:/var/lib/kubelet/pods/365a1938-752c-4cec-a917-c5a46c719307/volumes/kubernetes.io~projected/kube-api-access-qdv4k major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/365a1938-752c-4cec-a917-c5a46c719307/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/365a1938-752c-4cec-a917-c5a46c719307/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6498eed3-6528-4a49-a5d6-3dfcf9d75017/volumes/kubernetes.io~projected/kube-api-access-xgx8f:{mountpoint:/var/lib/kubelet/pods/6498eed3-6528-4a49-a5d6-3dfcf9d75017/volumes/kubernetes.io~projected/kube-api-access-xgx8f major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6813e008-7728-4466-979d-3ba72892a647/volumes/kubernetes.io~projected/kube-api-access-b6l77:{mountpoint:/var/lib/kubelet/pods/6813e008-7728-4466-979d-3ba72892a647/volumes/kubernetes.io~projected/kube-api-access-b6l77 major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8187068e-17ec-4e6d-bb8e-1d0312aa6674/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/8187068e-17ec-4e6d-bb8e-1d0312aa6674/volumes/kubernetes.io~secret/agent-certs major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/867a2a76-8aa3-4b07-98bf-913c2d2a5562/volumes/kubernetes.io~projected/kube-api-access-f6x5m:{mountpoint:/var/lib/kubelet/pods/867a2a76-8aa3-4b07-98bf-913c2d2a5562/volumes/kubernetes.io~projected/kube-api-access-f6x5m major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8ea0b360-1d72-4d40-935e-1394fecad74c/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/8ea0b360-1d72-4d40-935e-1394fecad74c/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/94868020-2e98-4295-8db5-043461cc6e9d/volumes/kubernetes.io~projected/kube-api-access-fswpg:{mountpoint:/var/lib/kubelet/pods/94868020-2e98-4295-8db5-043461cc6e9d/volumes/kubernetes.io~projected/kube-api-access-fswpg major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9b071df2-94c8-4459-89f2-f4b448ceb581/volumes/kubernetes.io~projected/kube-api-access-mhr4l:{mountpoint:/var/lib/kubelet/pods/9b071df2-94c8-4459-89f2-f4b448ceb581/volumes/kubernetes.io~projected/kube-api-access-mhr4l major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a06ec3e9-0a6e-46d8-ade5-13d657a6284c/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/a06ec3e9-0a6e-46d8-ade5-13d657a6284c/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a06ec3e9-0a6e-46d8-ade5-13d657a6284c/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/a06ec3e9-0a6e-46d8-ade5-13d657a6284c/volumes/kubernetes.io~empty-dir/tmp major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a06ec3e9-0a6e-46d8-ade5-13d657a6284c/volumes/kubernetes.io~projected/kube-api-access-t7l82:{mountpoint:/var/lib/kubelet/pods/a06ec3e9-0a6e-46d8-ade5-13d657a6284c/volumes/kubernetes.io~projected/kube-api-access-t7l82 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ac5ecdf9-3675-4d4f-98e3-a4e8796669be/volumes/kubernetes.io~projected/kube-api-access-png47:{mountpoint:/var/lib/kubelet/pods/ac5ecdf9-3675-4d4f-98e3-a4e8796669be/volumes/kubernetes.io~projected/kube-api-access-png47 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea/volumes/kubernetes.io~projected/kube-api-access-nrr2v:{mountpoint:/var/lib/kubelet/pods/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea/volumes/kubernetes.io~projected/kube-api-access-nrr2v major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cc66a3de-7e7e-4855-8f17-db97dcfa815a/volumes/kubernetes.io~projected/kube-api-access-2ssqq:{mountpoint:/var/lib/kubelet/pods/cc66a3de-7e7e-4855-8f17-db97dcfa815a/volumes/kubernetes.io~projected/kube-api-access-2ssqq major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fa1fd639-c265-4048-b9be-a583885ccb4e/volumes/kubernetes.io~projected/kube-api-access-shrkm:{mountpoint:/var/lib/kubelet/pods/fa1fd639-c265-4048-b9be-a583885ccb4e/volumes/kubernetes.io~projected/kube-api-access-shrkm major:0 minor:66 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/5bda1d5e1ece0e18aa718ebc7c909bd6ed0bfbf1e76da9c2f09c4c6c1ca18974/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/f847cf415548aa40ad8fd5db8647a847f58fca82e60d2a67383b3166e83e6883/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/9b5eb804c96912747d0ce6f71080b4a9f4bf72e9e7166687ed41817c3d8aceb2/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/c5f8e8a71d0e0b2985ae5fc8db2f329a4dfae9ab326622b71e32218f982c10bd/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/fb16d06fa6b4a0f6ce49d2801cbc188aec46d6b14b903e3a22107ff01ad454bc/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/a243474d705b32a6b333486475db2d103e820730be8e66da2ced717d14e719c6/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/5b001c44d93b97cf35416aac38e8536125d59dec8456dcbe9c07c0653fb3afc5/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/fdff52d108fb6d025a75765c964c8787b5942ca3a5b3c3aca829ccd48dd2a0ce/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/f50d4452724b58ce973c6484d5d82885d26d52baf3d0afbdf16474f820044b58/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/815bfe53bf9992471300cd74ebc341cb48f15aacc4066fc5f27edea8bdbecf1c/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/e8cd06533d8f399f83b10a8c8e449176984348e116334a04adfa318aa94207c0/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-149:{mountpoint:/var/lib/containers/storage/overlay/55fb9c64f2cedaacf8582186d89017cff9c2d5a697ecb077c953ea7452536a1b/merged major:0 minor:149 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/944531b8c282aa7305ed554314a1042871cb8a841ed7a3b3dee5b7954d6bde1b/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/b1892a4c1bd495ee75cc6c9ee65affc3b01918273fcd5f61bddd52415733962c/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-161:{mountpoint:/var/lib/containers/storage/overlay/be802c19da32763033356ce966c8aefd7d6acfe53f39b076e891207f0eec0929/merged major:0 minor:161 fsType:overlay blockSize:0} overlay_0-163:{mountpoint:/var/lib/containers/storage/overlay/3c10e70960bf9b72e777f22ce06bea8cb34544b1ea9c4f6b3341b5a7113bd6b6/merged major:0 minor:163 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/970a2bc59553918f027185252d911c6a4ee8c4549d79e9b38d4f1e8b4572fadc/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/2ba469436f56fbe8a1159b395de7276372546bb7186d3302afd1a551da26eff8/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/47e6f891deb081c51aace9172c3f4e6a595976eea5286ef29d4ed05b9380026b/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/426bd5dda505dbb583eb9d6db7c59ad3935d9b6d7cefd772725cb470ba7bfe29/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/496cf0a1e4b8184f200a07cac1a1af53135f3a42439ce0d3772a062b91a36191/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/999addf2fb2e675b99de43167f3d5f3cdf016e8d7628f366bba38b5016104463/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-207:{mountpoint:/var/lib/containers/storage/overlay/9104c1384978f604f920a17058518c4e322d875f4074981bd7cd7ed184dc54c1/merged major:0 minor:207 fsType:overlay blockSize:0} overlay_0-220:{mountpoint:/var/lib/containers/storage/overlay/3744182c7591c20afaf98e320dffecdc75382a86e8c3f8e0684ebd5567d4bd16/merged major:0 minor:220 fsType:overlay blockSize:0} overlay_0-224:{mountpoint:/var/lib/containers/storage/overlay/c50bc3a53f0af81aae063abdda7a57e88d34641136a1d8502d56f1ad01fa72f9/merged major:0 minor:224 fsType:overlay blockSize:0} overlay_0-226:{mountpoint:/var/lib/containers/storage/overlay/14cefff6e03b55b08286ec0675cdabbc31f54747ca088bd17ec6dd3213a02a5f/merged major:0 minor:226 fsType:overlay blockSize:0} overlay_0-228:{mountpoint:/var/lib/containers/storage/overlay/5d7e50b0a1431b2d1090d45a2e4c02e4f988a52ab580012c5f5fc6d9caa32a1a/merged major:0 minor:228 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/ed0fab8b1ea6ce2e4fdcad61ec5052e28274b3603d7432bcdbe740d4b7d713c4/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/399a445de9479ec0f6293ee115fc5b94781d2c9b25e802161f6811e19288419d/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/2f4f22efd033aec3d3e115dbb4560f8388bfc976603a497d484026e8b0d3154a/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/ce322968ffc99c909edb4b972320191be31e0d32ccc2f054b134b41fb69fe0d6/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/c4bef4447651bc4a902874a3a6df2a13a011712ca1dd8238f8db03ba9993acab/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/8006c9a2c96a1a38cb0ffef4233f237a784742d0d14ba9a2f555b1d45e6f012d/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/0b7144559f1ba55c14607755708d8ce406a9d5ddcd8ed79a1976e06b7952ece2/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/cde033f19e17b3ec403ebb2b41236f1d1db32fe379ce3cc90c798780e96ae4e6/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/4b8f3dab2f944b28f8615c26a0bd421a2620b7ea6d0ff68e909e8a65a6ad95d9/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/eca4464e21e0504e3a33a052db9722baa29d0640aa37dad821349c384f705eed/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/cc3d5cc8caf09fe82405a8b16d9cf137fc425331246b57c11bfdae386f0f2037/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/c1576d7adbb73670edc0f538734f4e02ad754d8de0123fe3097f5fa01721e6a6/merged major:0 minor:96 fsType:overlay blockSize:0}]
Nov 03 22:12:56.038537 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.035330    4431 manager.go:217] Machine: {Timestamp:2026-11-03 22:12:56.034850057 +0000 UTC m=+0.060705911 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3104311 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec206bb765ff4f0b7df103be4977e5fa SystemUUID:ec206bb7-65ff-4f0b-7df1-03be4977e5fa BootID:72dce58a-b349-428f-9b57-2111e098e01c Filesystems:[{Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a06ec3e9-0a6e-46d8-ade5-13d657a6284c/volumes/kubernetes.io~projected/kube-api-access-t7l82 DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7b9044ebe11612a0e9eded07aab7df7c563a18a145b86286b40ed006dfb27949/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/73a63c22ffae1c6c4abf49a45190081732a2518699b857e09e444501a9ed590a/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/8187068e-17ec-4e6d-bb8e-1d0312aa6674/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/69a840eb07f498030ab8307ebdd6e0e3f3dd7188385aa3f7dec3998e7b8b3c21/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/972ef7c557baa631f2303101cdfb55622fd8b8ceea4288d2f527e38c4bd4dfa8/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9427fc09e99178a5f76492a05fc10f074027d4022ad640cc3a535c624220f6d8/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/365a1938-752c-4cec-a917-c5a46c719307/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/365a1938-752c-4cec-a917-c5a46c719307/volumes/kubernetes.io~projected/kube-api-access-qdv4k DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-226 DeviceMajor:0 DeviceMinor:226 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea/volumes/kubernetes.io~projected/kube-api-access-nrr2v DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/537a5c4de9dae0664356e0290f054d99f3eefc22cc0b7682faa7cd0146fa2da5/userdata/shm DeviceMajor:0 DeviceMinor:210 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cbd1adc5425c7bbbc9dbc85c8ea9f67b3383c6657e7b8dc4ed932383257eed4a/userdata/shm DeviceMajor:0 DeviceMinor:222 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fa1fd639-c265-4048-b9be-a583885ccb4e/volumes/kubernetes.io~projected/kube-api-access-shrkm DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-220 DeviceMajor:0 DeviceMinor:220 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/867a2a76-8aa3-4b07-98bf-913c2d2a5562/volumes/kubernetes.io~projected/kube-api-access-f6x5m DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6498eed3-6528-4a49-a5d6-3dfcf9d75017/volumes/kubernetes.io~projected/kube-api-access-xgx8f DeviceMajor:0 DeviceMinor:209 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a06ec3e9-0a6e-46d8-ade5-13d657a6284c/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-163 DeviceMajor:0 DeviceMinor:163 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-228 DeviceMajor:0 DeviceMinor:228 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/05e93303e59fe467c6f7d583564effdd8a09180902c8db08e8211ba9ad1b3861/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-161 DeviceMajor:0 DeviceMinor:161 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/46377aeffc4cd3d572a6c7b722747c8c24bbebb5583a67d3a04edfb6df236e07/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9b071df2-94c8-4459-89f2-f4b448ceb581/volumes/kubernetes.io~projected/kube-api-access-mhr4l DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d02c882efc6b58702c439be61f776a0b522e52d436f895cce65cb3ff7d585d6b/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cc66a3de-7e7e-4855-8f17-db97dcfa815a/volumes/kubernetes.io~projected/kube-api-access-2ssqq DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/382c0ab10609a37b4a1f8e32d5dac3760c552c63c3a60682227fb3bfdaafca3c/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/309c2503-3fef-4048-af01-6b3b70aa52a1/volumes/kubernetes.io~projected/kube-api-access-8slxn DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6813e008-7728-4466-979d-3ba72892a647/volumes/kubernetes.io~projected/kube-api-access-b6l77 DeviceMajor:0 DeviceMinor:206 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-207 DeviceMajor:0 DeviceMinor:207 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a06ec3e9-0a6e-46d8-ade5-13d657a6284c/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ac5ecdf9-3675-4d4f-98e3-a4e8796669be/volumes/kubernetes.io~projected/kube-api-access-png47 DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5a9c9ef3d251d426e883ac5835699c7f59588d1b12d6f68ced725e4607dbdaf5/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-149 DeviceMajor:0 DeviceMinor:149 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1b792dc0-5e1c-40c4-a6c6-f68b4c6c8aac/volumes/kubernetes.io~projected/kube-api-access-qdtpv DeviceMajor:0 DeviceMinor:217 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/94868020-2e98-4295-8db5-043461cc6e9d/volumes/kubernetes.io~projected/kube-api-access-fswpg DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/01527eb9e50404bb5f91b695b61e185b6e7cf0927fd0d1252a2bb9d0c36ddc0e/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/8ea0b360-1d72-4d40-935e-1394fecad74c/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:215 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0452dadfd2655386a337e6ef379ca20341938653d5f53aa2545e56566fe7e56a/userdata/shm DeviceMajor:0 DeviceMinor:218 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-224 DeviceMajor:0 DeviceMinor:224 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/089f9a4b-8eec-4758-811b-e506b41cecbb/volumes/kubernetes.io~projected/kube-api-access-7bjhs DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0452dadfd265538 MacAddress:16:4a:6c:43:7f:a5 Speed:10000 Mtu:8901} {Name:537a5c4de9dae06 MacAddress:02:98:12:52:87:0e Speed:10000 Mtu:8901} {Name:73a63c22ffae1c6 MacAddress:ee:d3:c6:8c:a3:ff Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:de:44:3a:dd Speed:0 Mtu:9001} {Name:br-int MacAddress:26:49:7a:a6:88:0f Speed:0 Mtu:8901} {Name:cbd1adc5425c7bb MacAddress:5e:ca:ba:49:c5:81 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:de:44:3a:dd Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:d6:61:69:c9:d6:81 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:f6:9a:4d:ff:93:cf Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 22:12:56.038537 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.035590    4431 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 22:12:56.038537 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.035634    4431 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 22:12:56.038537 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.035902    4431 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 22:12:56.038537 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.035927    4431 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-73.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 22:12:56.038537 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.036061    4431 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 22:12:56.038537 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.036068    4431 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 22:12:56.038537 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.036078    4431 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 22:12:56.038537 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.036094    4431 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 22:12:56.038537 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.036186    4431 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 22:12:56.038537 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.036362    4431 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 22:12:56.038537 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.036471    4431 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 22:12:56.038537 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.036505    4431 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 22:12:56.038537 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.036518    4431 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 22:12:56.038537 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.036528    4431 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 22:12:56.038537 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.036537    4431 kubelet.go:405] "Adding apiserver pod source"
Nov 03 22:12:56.038537 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.036545    4431 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 22:12:56.038537 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.037115    4431 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 22:12:56.038537 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.037124    4431 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 22:12:56.038537 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.037599    4431 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 22:12:56.039450 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.039436    4431 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 22:12:56.039450 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.039449    4431 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 22:12:56.039547 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.039537    4431 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 03 22:12:56.039728 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.039718    4431 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 22:12:56.039764 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.039747    4431 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 22:12:56.039851 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.039843    4431 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 22:12:56.039884 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.039856    4431 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 22:12:56.039884 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.039863    4431 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 22:12:56.039884 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.039869    4431 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 22:12:56.039884 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.039875    4431 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 22:12:56.039884 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.039881    4431 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 22:12:56.040034 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.039887    4431 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 22:12:56.040034 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.039892    4431 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 22:12:56.040034 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.039899    4431 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 22:12:56.040034 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.039912    4431 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 22:12:56.040034 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.039924    4431 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 22:12:56.040034 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.039934    4431 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 22:12:56.040034 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.039970    4431 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 22:12:56.040034 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.039977    4431 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 22:12:56.043197 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.043182    4431 server.go:1263] "Started kubelet"
Nov 03 22:12:56.043406 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.043266    4431 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 22:12:56.043406 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.043342    4431 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 22:12:56.043507 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.043412    4431 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 22:12:56.043774 ip-10-0-139-73 systemd[1]: Started Kubernetes Kubelet.
Nov 03 22:12:56.044089 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.043801    4431 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 22:12:56.045118 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.045042    4431 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 22:12:56.046467 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.046449    4431 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 22:12:56.047309 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.047181    4431 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 22:12:56.047309 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.047257    4431 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 22:12:56.047309 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.047290    4431 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 22:12:56.047496 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.047413    4431 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 22:12:56.047651 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:12:56.047595    4431 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-73.ec2.internal\" not found"
Nov 03 22:12:56.047718 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.047661    4431 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 22:07:18 +0000 UTC" deadline="2028-06-07 06:31:35.813972859 +0000 UTC"
Nov 03 22:12:56.047718 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.047702    4431 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13952h18m39.766274975s"
Nov 03 22:12:56.049700 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.049660    4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ea0b360-1d72-4d40-935e-1394fecad74c" volumeName="kubernetes.io/secret/8ea0b360-1d72-4d40-935e-1394fecad74c-original-pull-secret" seLinuxMountContext=""
Nov 03 22:12:56.049792 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.049710    4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac5ecdf9-3675-4d4f-98e3-a4e8796669be" volumeName="kubernetes.io/projected/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-kube-api-access-png47" seLinuxMountContext=""
Nov 03 22:12:56.049792 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.049732    4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea" volumeName="kubernetes.io/configmap/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-ovnkube-script-lib" seLinuxMountContext=""
Nov 03 22:12:56.049792 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.049757    4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea" volumeName="kubernetes.io/projected/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-kube-api-access-nrr2v" seLinuxMountContext=""
Nov 03 22:12:56.049792 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.049772    4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc66a3de-7e7e-4855-8f17-db97dcfa815a" volumeName="kubernetes.io/configmap/cc66a3de-7e7e-4855-8f17-db97dcfa815a-serviceca" seLinuxMountContext=""
Nov 03 22:12:56.049985 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.049796    4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc66a3de-7e7e-4855-8f17-db97dcfa815a" volumeName="kubernetes.io/projected/cc66a3de-7e7e-4855-8f17-db97dcfa815a-kube-api-access-2ssqq" seLinuxMountContext=""
Nov 03 22:12:56.049985 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.049818    4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="089f9a4b-8eec-4758-811b-e506b41cecbb" volumeName="kubernetes.io/projected/089f9a4b-8eec-4758-811b-e506b41cecbb-kube-api-access-7bjhs" seLinuxMountContext=""
Nov 03 22:12:56.049985 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.049858    4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b792dc0-5e1c-40c4-a6c6-f68b4c6c8aac" volumeName="kubernetes.io/projected/1b792dc0-5e1c-40c4-a6c6-f68b4c6c8aac-kube-api-access-qdtpv" seLinuxMountContext=""
Nov 03 22:12:56.049985 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.049883    4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="309c2503-3fef-4048-af01-6b3b70aa52a1" volumeName="kubernetes.io/configmap/309c2503-3fef-4048-af01-6b3b70aa52a1-config-volume" seLinuxMountContext=""
Nov 03 22:12:56.049985 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.049901    4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="309c2503-3fef-4048-af01-6b3b70aa52a1" volumeName="kubernetes.io/empty-dir/309c2503-3fef-4048-af01-6b3b70aa52a1-tmp-dir" seLinuxMountContext=""
Nov 03 22:12:56.049985 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.049920    4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="365a1938-752c-4cec-a917-c5a46c719307" volumeName="kubernetes.io/projected/365a1938-752c-4cec-a917-c5a46c719307-kube-api-access-qdv4k" seLinuxMountContext=""
Nov 03 22:12:56.049985 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.049937    4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6813e008-7728-4466-979d-3ba72892a647" volumeName="kubernetes.io/empty-dir/6813e008-7728-4466-979d-3ba72892a647-data-volume" seLinuxMountContext=""
Nov 03 22:12:56.049985 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.049952    4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="867a2a76-8aa3-4b07-98bf-913c2d2a5562" volumeName="kubernetes.io/projected/867a2a76-8aa3-4b07-98bf-913c2d2a5562-kube-api-access-f6x5m" seLinuxMountContext=""
Nov 03 22:12:56.049985 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.049971    4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b071df2-94c8-4459-89f2-f4b448ceb581" volumeName="kubernetes.io/projected/9b071df2-94c8-4459-89f2-f4b448ceb581-kube-api-access-mhr4l" seLinuxMountContext=""
Nov 03 22:12:56.050409 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.049991    4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea" volumeName="kubernetes.io/configmap/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-env-overrides" seLinuxMountContext=""
Nov 03 22:12:56.050409 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.050016    4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="089f9a4b-8eec-4758-811b-e506b41cecbb" volumeName="kubernetes.io/configmap/089f9a4b-8eec-4758-811b-e506b41cecbb-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 03 22:12:56.050409 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.050039    4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="94868020-2e98-4295-8db5-043461cc6e9d" volumeName="kubernetes.io/empty-dir/94868020-2e98-4295-8db5-043461cc6e9d-tmp-dir" seLinuxMountContext=""
Nov 03 22:12:56.050409 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.050059    4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a06ec3e9-0a6e-46d8-ade5-13d657a6284c" volumeName="kubernetes.io/empty-dir/a06ec3e9-0a6e-46d8-ade5-13d657a6284c-tmp" seLinuxMountContext=""
Nov 03 22:12:56.050409 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.050075    4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="089f9a4b-8eec-4758-811b-e506b41cecbb" volumeName="kubernetes.io/configmap/089f9a4b-8eec-4758-811b-e506b41cecbb-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 03 22:12:56.050409 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.050088    4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6813e008-7728-4466-979d-3ba72892a647" volumeName="kubernetes.io/projected/6813e008-7728-4466-979d-3ba72892a647-kube-api-access-b6l77" seLinuxMountContext=""
Nov 03 22:12:56.050409 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.050107    4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8187068e-17ec-4e6d-bb8e-1d0312aa6674" volumeName="kubernetes.io/secret/8187068e-17ec-4e6d-bb8e-1d0312aa6674-agent-certs" seLinuxMountContext=""
Nov 03 22:12:56.050409 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.050124    4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac5ecdf9-3675-4d4f-98e3-a4e8796669be" volumeName="kubernetes.io/configmap/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-multus-daemon-config" seLinuxMountContext=""
Nov 03 22:12:56.050409 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.050150    4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="365a1938-752c-4cec-a917-c5a46c719307" volumeName="kubernetes.io/secret/365a1938-752c-4cec-a917-c5a46c719307-cert" seLinuxMountContext=""
Nov 03 22:12:56.050409 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.050164    4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac5ecdf9-3675-4d4f-98e3-a4e8796669be" volumeName="kubernetes.io/configmap/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-cni-binary-copy" seLinuxMountContext=""
Nov 03 22:12:56.050409 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.050177    4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6498eed3-6528-4a49-a5d6-3dfcf9d75017" volumeName="kubernetes.io/projected/6498eed3-6528-4a49-a5d6-3dfcf9d75017-kube-api-access-xgx8f" seLinuxMountContext=""
Nov 03 22:12:56.050409 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.050199    4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="94868020-2e98-4295-8db5-043461cc6e9d" volumeName="kubernetes.io/projected/94868020-2e98-4295-8db5-043461cc6e9d-kube-api-access-fswpg" seLinuxMountContext=""
Nov 03 22:12:56.050409 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.050226    4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a06ec3e9-0a6e-46d8-ade5-13d657a6284c" volumeName="kubernetes.io/projected/a06ec3e9-0a6e-46d8-ade5-13d657a6284c-kube-api-access-t7l82" seLinuxMountContext=""
Nov 03 22:12:56.050409 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.050250    4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="309c2503-3fef-4048-af01-6b3b70aa52a1" volumeName="kubernetes.io/projected/309c2503-3fef-4048-af01-6b3b70aa52a1-kube-api-access-8slxn" seLinuxMountContext=""
Nov 03 22:12:56.050409 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.050275    4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a06ec3e9-0a6e-46d8-ade5-13d657a6284c" volumeName="kubernetes.io/empty-dir/a06ec3e9-0a6e-46d8-ade5-13d657a6284c-etc-tuned" seLinuxMountContext=""
Nov 03 22:12:56.050409 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.050305    4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea" volumeName="kubernetes.io/configmap/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-ovnkube-config" seLinuxMountContext=""
Nov 03 22:12:56.050409 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.050318    4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea" volumeName="kubernetes.io/secret/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 03 22:12:56.050409 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.050338    4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fa1fd639-c265-4048-b9be-a583885ccb4e" volumeName="kubernetes.io/projected/fa1fd639-c265-4048-b9be-a583885ccb4e-kube-api-access-shrkm" seLinuxMountContext=""
Nov 03 22:12:56.050409 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.050355    4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="089f9a4b-8eec-4758-811b-e506b41cecbb" volumeName="kubernetes.io/configmap/089f9a4b-8eec-4758-811b-e506b41cecbb-cni-binary-copy" seLinuxMountContext=""
Nov 03 22:12:56.050409 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.050360    4431 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 22:12:56.050409 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.050375    4431 factory.go:55] Registering systemd factory
Nov 03 22:12:56.050409 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.050387    4431 factory.go:223] Registration of the systemd container factory successfully
Nov 03 22:12:56.051448 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.050375    4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8187068e-17ec-4e6d-bb8e-1d0312aa6674" volumeName="kubernetes.io/configmap/8187068e-17ec-4e6d-bb8e-1d0312aa6674-konnectivity-ca" seLinuxMountContext=""
Nov 03 22:12:56.051448 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.050475    4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="867a2a76-8aa3-4b07-98bf-913c2d2a5562" volumeName="kubernetes.io/configmap/867a2a76-8aa3-4b07-98bf-913c2d2a5562-iptables-alerter-script" seLinuxMountContext=""
Nov 03 22:12:56.051448 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.050723    4431 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 22:12:56.051448 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.050737    4431 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 22:12:56.051448 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.050850    4431 factory.go:153] Registering CRI-O factory
Nov 03 22:12:56.051448 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.050867    4431 factory.go:223] Registration of the crio container factory successfully
Nov 03 22:12:56.051448 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.050887    4431 factory.go:103] Registering Raw factory
Nov 03 22:12:56.051448 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.050902    4431 manager.go:1199] Started watching for new ooms in manager
Nov 03 22:12:56.051448 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.051328    4431 manager.go:319] Starting recovery of all containers
Nov 03 22:12:56.073525 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.073492    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mx5q8_ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea/ovn-acl-logging/0.log"
Nov 03 22:12:56.074620 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.074338    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mx5q8_ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea/ovn-acl-logging/0.log"
Nov 03 22:12:56.078314 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.078282    4431 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 22:12:56.079575 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.079553    4431 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 22:12:56.079689 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.079587    4431 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 22:12:56.079689 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.079607    4431 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 22:12:56.079689 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:12:56.079643    4431 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 03 22:12:56.088478 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.088452    4431 generic.go:356] "Generic (PLEG): container finished" podID="089f9a4b-8eec-4758-811b-e506b41cecbb" containerID="8cac1f1fb9d6b63898381f9e31a4e59ecd28999ea3a9f3c4a9f83a644f7a0051" exitCode=0
Nov 03 22:12:56.088478 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.088477    4431 generic.go:356] "Generic (PLEG): container finished" podID="089f9a4b-8eec-4758-811b-e506b41cecbb" containerID="9a5f82966fff3bdd089db9bb9ae7b8893c357f728b516ae00dc153d198a006a8" exitCode=0
Nov 03 22:12:56.088602 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.088489    4431 generic.go:356] "Generic (PLEG): container finished" podID="089f9a4b-8eec-4758-811b-e506b41cecbb" containerID="640f26bacadb38ff46b56466290f040fc7efba4e04872b3c8309eb8e0a8cd9c4" exitCode=0
Nov 03 22:12:56.088602 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.088498    4431 generic.go:356] "Generic (PLEG): container finished" podID="089f9a4b-8eec-4758-811b-e506b41cecbb" containerID="d0ae1b2e3b72d5460f0938d7cd8be1ff2cccac83c4abd7491f80b24e2123188d" exitCode=0
Nov 03 22:12:56.088602 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.088506    4431 generic.go:356] "Generic (PLEG): container finished" podID="089f9a4b-8eec-4758-811b-e506b41cecbb" containerID="42af517f7c4b4578b80c3649684ffe8c3779ef369b2d3abb6aeb7c5ee2ab53a5" exitCode=0
Nov 03 22:12:56.088602 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.088516    4431 generic.go:356] "Generic (PLEG): container finished" podID="089f9a4b-8eec-4758-811b-e506b41cecbb" containerID="1a0d137e44a33d2070392ea05e7146e8b051f19e723dacdaaf22b4d51b221b01" exitCode=0
Nov 03 22:12:56.100444 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.100406    4431 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 22:12:56.100676 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.100559    4431 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 22:12:56.100676 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.100600    4431 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 03 22:12:56.102259 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.102233    4431 generic.go:356] "Generic (PLEG): container finished" podID="9e2325705fcd5fefcf1f90504938970b" containerID="4218d9aba019bb13e1453000949158430ba6f312c71019cd678a3464a8f93fbe" exitCode=0
Nov 03 22:12:56.109244 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.109206    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mx5q8_ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea/ovn-acl-logging/0.log"
Nov 03 22:12:56.109590 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.109566    4431 generic.go:356] "Generic (PLEG): container finished" podID="ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea" containerID="9a59b13d1f08e177740c394e01192da89fd9050fe180a943a7019dae9e6e4eef" exitCode=1
Nov 03 22:12:56.113391 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.113372    4431 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 22:12:56.113478 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.113393    4431 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 22:12:56.120330 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.120315    4431 manager.go:324] Recovery completed
Nov 03 22:12:56.133815 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.133792    4431 cpu_manager.go:226] "Starting" policy="none"
Nov 03 22:12:56.133815 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.133807    4431 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 22:12:56.133952 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.133826    4431 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 22:12:56.134002 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.133974    4431 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 03 22:12:56.134062 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.133989    4431 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 03 22:12:56.134062 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.134013    4431 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 03 22:12:56.134062 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.134021    4431 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 03 22:12:56.134062 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.134029    4431 policy_none.go:50] "Start"
Nov 03 22:12:56.134062 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.134046    4431 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 22:12:56.134062 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.134059    4431 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 22:12:56.134364 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.134268    4431 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 03 22:12:56.134364 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.134280    4431 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 03 22:12:56.134364 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.134288    4431 policy_none.go:44] "Start"
Nov 03 22:12:56.138226 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.138181    4431 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 22:12:56.141250 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.141234    4431 manager.go:341] "Starting Device Plugin manager"
Nov 03 22:12:56.141870 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:12:56.141257    4431 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 22:12:56.141870 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.141866    4431 server.go:87] "Starting device plugin registration server"
Nov 03 22:12:56.142123 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.142102    4431 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 22:12:56.142123 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.142114    4431 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 22:12:56.142294 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.142226    4431 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 22:12:56.142360 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.142321    4431 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 22:12:56.142410 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.142369    4431 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 22:12:56.142410 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.142375    4431 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 22:12:56.142540 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.142408    4431 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T22:12:56.142359002Z","UUID":"0b36c64b-d596-4520-ba7d-83975ecef70b","Handler":null,"Name":"","Endpoint":""}
Nov 03 22:12:56.143607 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:12:56.143583    4431 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 22:12:56.144141 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.144120    4431 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 22:12:56.144236 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.144149    4431 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 22:12:56.180248 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.180187    4431 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-73.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-73.ec2.internal"]
Nov 03 22:12:56.180822 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.180776    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-73.ec2.internal" event={"ID":"a7264ab43e0f926a44edf6ae184aafb2","Type":"ContainerStarted","Data":"5d6dd3ebaab3f6b7146fb4137385e43916a5fdd275c604e8d710477b733185de"}
Nov 03 22:12:56.180922 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.180828    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-73.ec2.internal" event={"ID":"a7264ab43e0f926a44edf6ae184aafb2","Type":"ContainerStarted","Data":"972ef7c557baa631f2303101cdfb55622fd8b8ceea4288d2f527e38c4bd4dfa8"}
Nov 03 22:12:56.180922 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.180848    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-73.ec2.internal" event={"ID":"9e2325705fcd5fefcf1f90504938970b","Type":"ContainerStarted","Data":"e52b79bc73f4045b5af1f640cbeb45337d0274daa92a7a674e6962cd198cb6e5"}
Nov 03 22:12:56.180922 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.180867    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-73.ec2.internal" event={"ID":"9e2325705fcd5fefcf1f90504938970b","Type":"ContainerDied","Data":"4218d9aba019bb13e1453000949158430ba6f312c71019cd678a3464a8f93fbe"}
Nov 03 22:12:56.180922 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.180887    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-73.ec2.internal" event={"ID":"9e2325705fcd5fefcf1f90504938970b","Type":"ContainerStarted","Data":"05e93303e59fe467c6f7d583564effdd8a09180902c8db08e8211ba9ad1b3861"}
Nov 03 22:12:56.181049 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.180923    4431 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-73.ec2.internal"
Nov 03 22:12:56.181049 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.180954    4431 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-73.ec2.internal"
Nov 03 22:12:56.204068 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.204046    4431 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 22:12:56.204166 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:12:56.204086    4431 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-139-73.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-73.ec2.internal"
Nov 03 22:12:56.204639 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.204624    4431 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 22:12:56.204673 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:12:56.204658    4431 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-139-73.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-73.ec2.internal"
Nov 03 22:12:56.243093 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.243069    4431 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 22:12:56.244400 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.244380    4431 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-73.ec2.internal" event="NodeHasSufficientMemory"
Nov 03 22:12:56.244483 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.244423    4431 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-73.ec2.internal" event="NodeHasNoDiskPressure"
Nov 03 22:12:56.244483 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.244438    4431 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-73.ec2.internal" event="NodeHasSufficientPID"
Nov 03 22:12:56.244483 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.244478    4431 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-73.ec2.internal"
Nov 03 22:12:56.279430 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.279404    4431 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-139-73.ec2.internal"
Nov 03 22:12:56.279544 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.279467    4431 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-139-73.ec2.internal"
Nov 03 22:12:56.302320 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.302294    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9e2325705fcd5fefcf1f90504938970b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-73.ec2.internal\" (UID: \"9e2325705fcd5fefcf1f90504938970b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-73.ec2.internal"
Nov 03 22:12:56.302430 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.302323    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e2325705fcd5fefcf1f90504938970b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-73.ec2.internal\" (UID: \"9e2325705fcd5fefcf1f90504938970b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-73.ec2.internal"
Nov 03 22:12:56.302430 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.302343    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a7264ab43e0f926a44edf6ae184aafb2-config\") pod \"kube-apiserver-proxy-ip-10-0-139-73.ec2.internal\" (UID: \"a7264ab43e0f926a44edf6ae184aafb2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-73.ec2.internal"
Nov 03 22:12:56.402993 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.402910    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9e2325705fcd5fefcf1f90504938970b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-73.ec2.internal\" (UID: \"9e2325705fcd5fefcf1f90504938970b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-73.ec2.internal"
Nov 03 22:12:56.402993 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.402863    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9e2325705fcd5fefcf1f90504938970b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-73.ec2.internal\" (UID: \"9e2325705fcd5fefcf1f90504938970b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-73.ec2.internal"
Nov 03 22:12:56.402993 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.402973    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e2325705fcd5fefcf1f90504938970b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-73.ec2.internal\" (UID: \"9e2325705fcd5fefcf1f90504938970b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-73.ec2.internal"
Nov 03 22:12:56.402993 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.402996    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a7264ab43e0f926a44edf6ae184aafb2-config\") pod \"kube-apiserver-proxy-ip-10-0-139-73.ec2.internal\" (UID: \"a7264ab43e0f926a44edf6ae184aafb2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-73.ec2.internal"
Nov 03 22:12:56.403234 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.403022    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a7264ab43e0f926a44edf6ae184aafb2-config\") pod \"kube-apiserver-proxy-ip-10-0-139-73.ec2.internal\" (UID: \"a7264ab43e0f926a44edf6ae184aafb2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-73.ec2.internal"
Nov 03 22:12:56.403234 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:56.403037    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e2325705fcd5fefcf1f90504938970b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-73.ec2.internal\" (UID: \"9e2325705fcd5fefcf1f90504938970b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-73.ec2.internal"
Nov 03 22:12:57.037437 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.037404    4431 apiserver.go:52] "Watching apiserver"
Nov 03 22:12:57.077936 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.077912    4431 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 22:12:57.078373 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.078352    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-x7cpm","kube-system/kube-apiserver-proxy-ip-10-0-139-73.ec2.internal","openshift-ingress-canary/ingress-canary-vhccw","kube-system/global-pull-secret-syncer-66m8v","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-73.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zcr2","openshift-ovn-kubernetes/ovnkube-node-mx5q8","openshift-network-diagnostics/network-check-target-82h4r","openshift-image-registry/node-ca-p4p25","openshift-kube-storage-version-migrator/migrator-66df5d496b-8dbwq","openshift-dns/dns-default-4qrls","openshift-insights/insights-runtime-extractor-q6ndx","openshift-network-operator/iptables-alerter-7rqgq","openshift-multus/network-metrics-daemon-jn5wf","openshift-multus/multus-2k9fs","openshift-multus/multus-additional-cni-plugins-4sm9h","kube-system/konnectivity-agent-7p8pl","openshift-cluster-node-tuning-operator/tuned-9ltv9"]
Nov 03 22:12:57.079817 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.079781    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jn5wf"
Nov 03 22:12:57.085512 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.085488    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4qrls"
Nov 03 22:12:57.086295 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.086277    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-q6ndx"
Nov 03 22:12:57.088202 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.088187    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-p4p25_cc66a3de-7e7e-4855-8f17-db97dcfa815a/node-ca/0.log"
Nov 03 22:12:57.094152 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.094131    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 22:12:57.094255 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.094150    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 22:12:57.094255 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.094164    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 22:12:57.094255 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.094183    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-mdchn\""
Nov 03 22:12:57.094255 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.094239    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 22:12:57.094457 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.094443    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zdncz\""
Nov 03 22:12:57.094870 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.094833    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 22:12:57.094870 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.094846    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 22:12:57.095248 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.095233    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 22:12:57.105595 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.105581    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rz46p\""
Nov 03 22:12:57.116910 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.116882    4431 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-73.ec2.internal"
Nov 03 22:12:57.117023 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.117009    4431 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-73.ec2.internal"
Nov 03 22:12:57.139665 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.139639    4431 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 22:12:57.139771 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:12:57.139685    4431 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-139-73.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-73.ec2.internal"
Nov 03 22:12:57.140073 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.140060    4431 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 22:12:57.140112 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:12:57.140093    4431 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-139-73.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-73.ec2.internal"
Nov 03 22:12:57.149306 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.149288    4431 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 22:12:57.192737 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.192703    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:12:57.195450 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.195422    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 22:12:57.195634 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.195606    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 22:12:57.195722 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.195653    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 22:12:57.195800 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.195784    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 22:12:57.195859 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.195804    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 22:12:57.195859 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.195832    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 22:12:57.195859 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.195821    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 22:12:57.195993 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.195867    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 22:12:57.195993 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.195904    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 22:12:57.195993 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.195915    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 22:12:57.196135 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.196030    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 22:12:57.196209 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.196193    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-zwzw5\""
Nov 03 22:12:57.196275 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.196208    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 03 22:12:57.196275 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.196249    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 22:12:57.196275 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.196271    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 22:12:57.196416 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.196302    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 22:12:57.196416 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.196359    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 03 22:12:57.196416 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.196393    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 22:12:57.196541 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.196420    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 22:12:57.196541 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.196427    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 22:12:57.196541 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.196423    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 22:12:57.196541 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.196431    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 22:12:57.196541 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.196448    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-wvrhf\""
Nov 03 22:12:57.196541 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.196460    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 22:12:57.196541 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.196481    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4tpvw\""
Nov 03 22:12:57.196541 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.196484    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 22:12:57.196541 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.196496    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 22:12:57.196541 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.196483    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7shc8\""
Nov 03 22:12:57.196541 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.196517    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-svbdj\""
Nov 03 22:12:57.196541 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.196521    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 22:12:57.196541 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.196484    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 22:12:57.196541 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.196526    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 22:12:57.196541 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.196496    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 22:12:57.196541 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.196542    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 22:12:57.197122 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.196545    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-h6vzq\""
Nov 03 22:12:57.197122 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.196644    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 22:12:57.197122 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.196672    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 22:12:57.197122 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.196680    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 22:12:57.197122 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.196726    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-wf9zk\""
Nov 03 22:12:57.197122 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.196759    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-nt8s4\""
Nov 03 22:12:57.197122 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.196812    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-mq5fv\""
Nov 03 22:12:57.197122 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.196916    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-sjknz\""
Nov 03 22:12:57.197122 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.196953    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-9wkj9\""
Nov 03 22:12:57.197122 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.197038    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-jwdqh\""
Nov 03 22:12:57.207099 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.207077    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-systemd-units\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:57.207151 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.207104    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a06ec3e9-0a6e-46d8-ade5-13d657a6284c-etc-sysctl-conf\") pod \"tuned-9ltv9\" (UID: \"a06ec3e9-0a6e-46d8-ade5-13d657a6284c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ltv9"
Nov 03 22:12:57.207151 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.207118    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a06ec3e9-0a6e-46d8-ade5-13d657a6284c-lib-modules\") pod \"tuned-9ltv9\" (UID: \"a06ec3e9-0a6e-46d8-ade5-13d657a6284c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ltv9"
Nov 03 22:12:57.207151 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.207132    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/089f9a4b-8eec-4758-811b-e506b41cecbb-cnibin\") pod \"multus-additional-cni-plugins-4sm9h\" (UID: \"089f9a4b-8eec-4758-811b-e506b41cecbb\") " pod="openshift-multus/multus-additional-cni-plugins-4sm9h"
Nov 03 22:12:57.207285 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.207150    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-run-ovn\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:57.207285 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.207198    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-log-socket\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:57.207285 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.207247    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a06ec3e9-0a6e-46d8-ade5-13d657a6284c-sys\") pod \"tuned-9ltv9\" (UID: \"a06ec3e9-0a6e-46d8-ade5-13d657a6284c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ltv9"
Nov 03 22:12:57.207410 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.207291    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8slxn\" (UniqueName: \"kubernetes.io/projected/309c2503-3fef-4048-af01-6b3b70aa52a1-kube-api-access-8slxn\") pod \"dns-default-4qrls\" (UID: \"309c2503-3fef-4048-af01-6b3b70aa52a1\") " pod="openshift-dns/dns-default-4qrls"
Nov 03 22:12:57.207410 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.207331    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94868020-2e98-4295-8db5-043461cc6e9d-tmp-dir\") pod \"node-resolver-x7cpm\" (UID: \"94868020-2e98-4295-8db5-043461cc6e9d\") " pod="openshift-dns/node-resolver-x7cpm"
Nov 03 22:12:57.207546 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.207357    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhr4l\" (UniqueName: \"kubernetes.io/projected/9b071df2-94c8-4459-89f2-f4b448ceb581-kube-api-access-mhr4l\") pod \"network-metrics-daemon-jn5wf\" (UID: \"9b071df2-94c8-4459-89f2-f4b448ceb581\") " pod="openshift-multus/network-metrics-daemon-jn5wf"
Nov 03 22:12:57.207808 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.207771    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/089f9a4b-8eec-4758-811b-e506b41cecbb-cni-binary-copy\") pod \"multus-additional-cni-plugins-4sm9h\" (UID: \"089f9a4b-8eec-4758-811b-e506b41cecbb\") " pod="openshift-multus/multus-additional-cni-plugins-4sm9h"
Nov 03 22:12:57.207808 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.207791    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94868020-2e98-4295-8db5-043461cc6e9d-tmp-dir\") pod \"node-resolver-x7cpm\" (UID: \"94868020-2e98-4295-8db5-043461cc6e9d\") " pod="openshift-dns/node-resolver-x7cpm"
Nov 03 22:12:57.208008 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.207901    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 22:12:57.208008 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.207916    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8ea0b360-1d72-4d40-935e-1394fecad74c-kubelet-config\") pod \"global-pull-secret-syncer-66m8v\" (UID: \"8ea0b360-1d72-4d40-935e-1394fecad74c\") " pod="kube-system/global-pull-secret-syncer-66m8v"
Nov 03 22:12:57.208008 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.207939    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-h4jdd\""
Nov 03 22:12:57.208008 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.207984    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fa1fd639-c265-4048-b9be-a583885ccb4e-socket-dir\") pod \"aws-ebs-csi-driver-node-8zcr2\" (UID: \"fa1fd639-c265-4048-b9be-a583885ccb4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zcr2"
Nov 03 22:12:57.208254 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.208164    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 22:12:57.208315 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.208253    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-multus-cni-dir\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:57.208534 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.208494    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-host-var-lib-cni-bin\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:57.208618 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.208564    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-multus-conf-dir\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:57.208618 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.208603    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/309c2503-3fef-4048-af01-6b3b70aa52a1-metrics-tls\") pod \"dns-default-4qrls\" (UID: \"309c2503-3fef-4048-af01-6b3b70aa52a1\") " pod="openshift-dns/dns-default-4qrls"
Nov 03 22:12:57.208722 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.208670    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-run-openvswitch\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:57.208722 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.208710    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-node-log\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:57.208820 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.208771    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/365a1938-752c-4cec-a917-c5a46c719307-cert\") pod \"ingress-canary-vhccw\" (UID: \"365a1938-752c-4cec-a917-c5a46c719307\") " pod="openshift-ingress-canary/ingress-canary-vhccw"
Nov 03 22:12:57.208974 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.208814    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-multus-socket-dir-parent\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:57.208974 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.208858    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fswpg\" (UniqueName: \"kubernetes.io/projected/94868020-2e98-4295-8db5-043461cc6e9d-kube-api-access-fswpg\") pod \"node-resolver-x7cpm\" (UID: \"94868020-2e98-4295-8db5-043461cc6e9d\") " pod="openshift-dns/node-resolver-x7cpm"
Nov 03 22:12:57.208974 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.208901    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cc66a3de-7e7e-4855-8f17-db97dcfa815a-host\") pod \"node-ca-p4p25\" (UID: \"cc66a3de-7e7e-4855-8f17-db97dcfa815a\") " pod="openshift-image-registry/node-ca-p4p25"
Nov 03 22:12:57.208974 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.208933    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/089f9a4b-8eec-4758-811b-e506b41cecbb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4sm9h\" (UID: \"089f9a4b-8eec-4758-811b-e506b41cecbb\") " pod="openshift-multus/multus-additional-cni-plugins-4sm9h"
Nov 03 22:12:57.208974 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.208937    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 22:12:57.208974 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.208966    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/089f9a4b-8eec-4758-811b-e506b41cecbb-cni-binary-copy\") pod \"multus-additional-cni-plugins-4sm9h\" (UID: \"089f9a4b-8eec-4758-811b-e506b41cecbb\") " pod="openshift-multus/multus-additional-cni-plugins-4sm9h"
Nov 03 22:12:57.209297 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.208961    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-host-run-netns\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:57.209297 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.209076    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fa1fd639-c265-4048-b9be-a583885ccb4e-sys-fs\") pod \"aws-ebs-csi-driver-node-8zcr2\" (UID: \"fa1fd639-c265-4048-b9be-a583885ccb4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zcr2"
Nov 03 22:12:57.209400 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.209339    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fa1fd639-c265-4048-b9be-a583885ccb4e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8zcr2\" (UID: \"fa1fd639-c265-4048-b9be-a583885ccb4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zcr2"
Nov 03 22:12:57.209591 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.209554    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fa1fd639-c265-4048-b9be-a583885ccb4e-etc-selinux\") pod \"aws-ebs-csi-driver-node-8zcr2\" (UID: \"fa1fd639-c265-4048-b9be-a583885ccb4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zcr2"
Nov 03 22:12:57.209698 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.209631    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-host-var-lib-kubelet\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:57.209698 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.209661    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6x5m\" (UniqueName: \"kubernetes.io/projected/867a2a76-8aa3-4b07-98bf-913c2d2a5562-kube-api-access-f6x5m\") pod \"iptables-alerter-7rqgq\" (UID: \"867a2a76-8aa3-4b07-98bf-913c2d2a5562\") " pod="openshift-network-operator/iptables-alerter-7rqgq"
Nov 03 22:12:57.209816 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.209723    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/089f9a4b-8eec-4758-811b-e506b41cecbb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4sm9h\" (UID: \"089f9a4b-8eec-4758-811b-e506b41cecbb\") " pod="openshift-multus/multus-additional-cni-plugins-4sm9h"
Nov 03 22:12:57.209816 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.209721    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6813e008-7728-4466-979d-3ba72892a647-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q6ndx\" (UID: \"6813e008-7728-4466-979d-3ba72892a647\") " pod="openshift-insights/insights-runtime-extractor-q6ndx"
Nov 03 22:12:57.210106 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.210088    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/94868020-2e98-4295-8db5-043461cc6e9d-hosts-file\") pod \"node-resolver-x7cpm\" (UID: \"94868020-2e98-4295-8db5-043461cc6e9d\") " pod="openshift-dns/node-resolver-x7cpm"
Nov 03 22:12:57.210195 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.210143    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-host-run-netns\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:57.210195 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.210176    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-host-cni-bin\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:57.210326 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.210200    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-host-cni-netd\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:57.210326 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.210283    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a06ec3e9-0a6e-46d8-ade5-13d657a6284c-tmp\") pod \"tuned-9ltv9\" (UID: \"a06ec3e9-0a6e-46d8-ade5-13d657a6284c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ltv9"
Nov 03 22:12:57.210448 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.210430    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7l82\" (UniqueName: \"kubernetes.io/projected/a06ec3e9-0a6e-46d8-ade5-13d657a6284c-kube-api-access-t7l82\") pod \"tuned-9ltv9\" (UID: \"a06ec3e9-0a6e-46d8-ade5-13d657a6284c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ltv9"
Nov 03 22:12:57.210528 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.210480    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-host-run-multus-certs\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:57.210598 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.210516    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6813e008-7728-4466-979d-3ba72892a647-crio-socket\") pod \"insights-runtime-extractor-q6ndx\" (UID: \"6813e008-7728-4466-979d-3ba72892a647\") " pod="openshift-insights/insights-runtime-extractor-q6ndx"
Nov 03 22:12:57.210598 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.210580    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6813e008-7728-4466-979d-3ba72892a647-data-volume\") pod \"insights-runtime-extractor-q6ndx\" (UID: \"6813e008-7728-4466-979d-3ba72892a647\") " pod="openshift-insights/insights-runtime-extractor-q6ndx"
Nov 03 22:12:57.210723 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.210613    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgx8f\" (UniqueName: \"kubernetes.io/projected/6498eed3-6528-4a49-a5d6-3dfcf9d75017-kube-api-access-xgx8f\") pod \"migrator-66df5d496b-8dbwq\" (UID: \"6498eed3-6528-4a49-a5d6-3dfcf9d75017\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8dbwq"
Nov 03 22:12:57.210723 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.210646    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-run-systemd\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:57.210723 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.210673    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a06ec3e9-0a6e-46d8-ade5-13d657a6284c-etc-modprobe-d\") pod \"tuned-9ltv9\" (UID: \"a06ec3e9-0a6e-46d8-ade5-13d657a6284c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ltv9"
Nov 03 22:12:57.210930 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.210723    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a06ec3e9-0a6e-46d8-ade5-13d657a6284c-etc-systemd\") pod \"tuned-9ltv9\" (UID: \"a06ec3e9-0a6e-46d8-ade5-13d657a6284c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ltv9"
Nov 03 22:12:57.210930 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.210786    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8187068e-17ec-4e6d-bb8e-1d0312aa6674-konnectivity-ca\") pod \"konnectivity-agent-7p8pl\" (UID: \"8187068e-17ec-4e6d-bb8e-1d0312aa6674\") " pod="kube-system/konnectivity-agent-7p8pl"
Nov 03 22:12:57.210930 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.210820    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-multus-daemon-config\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:57.210930 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.210848    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/309c2503-3fef-4048-af01-6b3b70aa52a1-tmp-dir\") pod \"dns-default-4qrls\" (UID: \"309c2503-3fef-4048-af01-6b3b70aa52a1\") " pod="openshift-dns/dns-default-4qrls"
Nov 03 22:12:57.211314 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.211283    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6813e008-7728-4466-979d-3ba72892a647-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q6ndx\" (UID: \"6813e008-7728-4466-979d-3ba72892a647\") " pod="openshift-insights/insights-runtime-extractor-q6ndx"
Nov 03 22:12:57.211417 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.211327    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cc66a3de-7e7e-4855-8f17-db97dcfa815a-serviceca\") pod \"node-ca-p4p25\" (UID: \"cc66a3de-7e7e-4855-8f17-db97dcfa815a\") " pod="openshift-image-registry/node-ca-p4p25"
Nov 03 22:12:57.211496 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.211421    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/089f9a4b-8eec-4758-811b-e506b41cecbb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4sm9h\" (UID: \"089f9a4b-8eec-4758-811b-e506b41cecbb\") " pod="openshift-multus/multus-additional-cni-plugins-4sm9h"
Nov 03 22:12:57.211587 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.211561    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a06ec3e9-0a6e-46d8-ade5-13d657a6284c-etc-sysctl-d\") pod \"tuned-9ltv9\" (UID: \"a06ec3e9-0a6e-46d8-ade5-13d657a6284c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ltv9"
Nov 03 22:12:57.211654 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.211577    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a06ec3e9-0a6e-46d8-ade5-13d657a6284c-tmp\") pod \"tuned-9ltv9\" (UID: \"a06ec3e9-0a6e-46d8-ade5-13d657a6284c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ltv9"
Nov 03 22:12:57.211809 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.211762    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-hostroot\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:57.211936 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.211804    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8187068e-17ec-4e6d-bb8e-1d0312aa6674-konnectivity-ca\") pod \"konnectivity-agent-7p8pl\" (UID: \"8187068e-17ec-4e6d-bb8e-1d0312aa6674\") " pod="kube-system/konnectivity-agent-7p8pl"
Nov 03 22:12:57.211936 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.211885    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-etc-kubernetes\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:57.212036 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.211934    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/309c2503-3fef-4048-af01-6b3b70aa52a1-config-volume\") pod \"dns-default-4qrls\" (UID: \"309c2503-3fef-4048-af01-6b3b70aa52a1\") " pod="openshift-dns/dns-default-4qrls"
Nov 03 22:12:57.212036 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.211932    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/365a1938-752c-4cec-a917-c5a46c719307-cert\") pod \"ingress-canary-vhccw\" (UID: \"365a1938-752c-4cec-a917-c5a46c719307\") " pod="openshift-ingress-canary/ingress-canary-vhccw"
Nov 03 22:12:57.212132 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.212037    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shrkm\" (UniqueName: \"kubernetes.io/projected/fa1fd639-c265-4048-b9be-a583885ccb4e-kube-api-access-shrkm\") pod \"aws-ebs-csi-driver-node-8zcr2\" (UID: \"fa1fd639-c265-4048-b9be-a583885ccb4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zcr2"
Nov 03 22:12:57.212233 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.212189    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cc66a3de-7e7e-4855-8f17-db97dcfa815a-serviceca\") pod \"node-ca-p4p25\" (UID: \"cc66a3de-7e7e-4855-8f17-db97dcfa815a\") " pod="openshift-image-registry/node-ca-p4p25"
Nov 03 22:12:57.212387 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.212339    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:57.212471 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.212412    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/309c2503-3fef-4048-af01-6b3b70aa52a1-config-volume\") pod \"dns-default-4qrls\" (UID: \"309c2503-3fef-4048-af01-6b3b70aa52a1\") " pod="openshift-dns/dns-default-4qrls"
Nov 03 22:12:57.212536 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.212451    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrr2v\" (UniqueName: \"kubernetes.io/projected/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-kube-api-access-nrr2v\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:57.212536 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.211912    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/089f9a4b-8eec-4758-811b-e506b41cecbb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4sm9h\" (UID: \"089f9a4b-8eec-4758-811b-e506b41cecbb\") " pod="openshift-multus/multus-additional-cni-plugins-4sm9h"
Nov 03 22:12:57.212536 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.212500    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/309c2503-3fef-4048-af01-6b3b70aa52a1-tmp-dir\") pod \"dns-default-4qrls\" (UID: \"309c2503-3fef-4048-af01-6b3b70aa52a1\") " pod="openshift-dns/dns-default-4qrls"
Nov 03 22:12:57.212536 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.212529    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a06ec3e9-0a6e-46d8-ade5-13d657a6284c-host\") pod \"tuned-9ltv9\" (UID: \"a06ec3e9-0a6e-46d8-ade5-13d657a6284c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ltv9"
Nov 03 22:12:57.212734 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.212568    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a06ec3e9-0a6e-46d8-ade5-13d657a6284c-etc-kubernetes\") pod \"tuned-9ltv9\" (UID: \"a06ec3e9-0a6e-46d8-ade5-13d657a6284c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ltv9"
Nov 03 22:12:57.212734 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.212578    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-multus-daemon-config\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:57.212734 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.212625    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a06ec3e9-0a6e-46d8-ade5-13d657a6284c-var-lib-kubelet\") pod \"tuned-9ltv9\" (UID: \"a06ec3e9-0a6e-46d8-ade5-13d657a6284c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ltv9"
Nov 03 22:12:57.212734 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.212678    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a06ec3e9-0a6e-46d8-ade5-13d657a6284c-etc-tuned\") pod \"tuned-9ltv9\" (UID: \"a06ec3e9-0a6e-46d8-ade5-13d657a6284c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ltv9"
Nov 03 22:12:57.212734 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.212688    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6813e008-7728-4466-979d-3ba72892a647-data-volume\") pod \"insights-runtime-extractor-q6ndx\" (UID: \"6813e008-7728-4466-979d-3ba72892a647\") " pod="openshift-insights/insights-runtime-extractor-q6ndx"
Nov 03 22:12:57.212734 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.212731    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6l77\" (UniqueName: \"kubernetes.io/projected/6813e008-7728-4466-979d-3ba72892a647-kube-api-access-b6l77\") pod \"insights-runtime-extractor-q6ndx\" (UID: \"6813e008-7728-4466-979d-3ba72892a647\") " pod="openshift-insights/insights-runtime-extractor-q6ndx"
Nov 03 22:12:57.213001 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.212787    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a06ec3e9-0a6e-46d8-ade5-13d657a6284c-etc-tuned\") pod \"tuned-9ltv9\" (UID: \"a06ec3e9-0a6e-46d8-ade5-13d657a6284c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ltv9"
Nov 03 22:12:57.213001 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.212796    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-host-slash\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:57.213001 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.212879    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a06ec3e9-0a6e-46d8-ade5-13d657a6284c-etc-sysconfig\") pod \"tuned-9ltv9\" (UID: \"a06ec3e9-0a6e-46d8-ade5-13d657a6284c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ltv9"
Nov 03 22:12:57.213001 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.212921    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-cnibin\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:57.213001 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.212963    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/867a2a76-8aa3-4b07-98bf-913c2d2a5562-host-slash\") pod \"iptables-alerter-7rqgq\" (UID: \"867a2a76-8aa3-4b07-98bf-913c2d2a5562\") " pod="openshift-network-operator/iptables-alerter-7rqgq"
Nov 03 22:12:57.213186 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.213002    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ssqq\" (UniqueName: \"kubernetes.io/projected/cc66a3de-7e7e-4855-8f17-db97dcfa815a-kube-api-access-2ssqq\") pod \"node-ca-p4p25\" (UID: \"cc66a3de-7e7e-4855-8f17-db97dcfa815a\") " pod="openshift-image-registry/node-ca-p4p25"
Nov 03 22:12:57.213186 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.213045    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8ea0b360-1d72-4d40-935e-1394fecad74c-dbus\") pod \"global-pull-secret-syncer-66m8v\" (UID: \"8ea0b360-1d72-4d40-935e-1394fecad74c\") " pod="kube-system/global-pull-secret-syncer-66m8v"
Nov 03 22:12:57.213465 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.213446    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-var-lib-openvswitch\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:57.213529 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.213512    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-env-overrides\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:57.213574 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.213557    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a06ec3e9-0a6e-46d8-ade5-13d657a6284c-run\") pod \"tuned-9ltv9\" (UID: \"a06ec3e9-0a6e-46d8-ade5-13d657a6284c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ltv9"
Nov 03 22:12:57.213646 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.213629    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-host-run-k8s-cni-cncf-io\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:57.213692 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.213668    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-png47\" (UniqueName: \"kubernetes.io/projected/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-kube-api-access-png47\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:57.213731 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.213719    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-env-overrides\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:57.213840 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.213819    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/867a2a76-8aa3-4b07-98bf-913c2d2a5562-iptables-alerter-script\") pod \"iptables-alerter-7rqgq\" (UID: \"867a2a76-8aa3-4b07-98bf-913c2d2a5562\") " pod="openshift-network-operator/iptables-alerter-7rqgq"
Nov 03 22:12:57.213875 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.213856    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-etc-openvswitch\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:57.213921 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.213905    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-host-run-ovn-kubernetes\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:57.213973 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.213938    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-ovn-node-metrics-cert\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:57.214014 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.213982    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-ovnkube-script-lib\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:57.214097 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.214079    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/867a2a76-8aa3-4b07-98bf-913c2d2a5562-iptables-alerter-script\") pod \"iptables-alerter-7rqgq\" (UID: \"867a2a76-8aa3-4b07-98bf-913c2d2a5562\") " pod="openshift-network-operator/iptables-alerter-7rqgq"
Nov 03 22:12:57.214160 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.214099    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-ovn-node-metrics-cert\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:57.214160 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.214102    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8187068e-17ec-4e6d-bb8e-1d0312aa6674-agent-certs\") pod \"konnectivity-agent-7p8pl\" (UID: \"8187068e-17ec-4e6d-bb8e-1d0312aa6674\") " pod="kube-system/konnectivity-agent-7p8pl"
Nov 03 22:12:57.214160 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.214145    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fa1fd639-c265-4048-b9be-a583885ccb4e-registration-dir\") pod \"aws-ebs-csi-driver-node-8zcr2\" (UID: \"fa1fd639-c265-4048-b9be-a583885ccb4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zcr2"
Nov 03 22:12:57.214326 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.214161    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-ovnkube-config\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:57.214326 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.214268    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdtpv\" (UniqueName: \"kubernetes.io/projected/1b792dc0-5e1c-40c4-a6c6-f68b4c6c8aac-kube-api-access-qdtpv\") pod \"network-check-target-82h4r\" (UID: \"1b792dc0-5e1c-40c4-a6c6-f68b4c6c8aac\") " pod="openshift-network-diagnostics/network-check-target-82h4r"
Nov 03 22:12:57.214326 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.214289    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdv4k\" (UniqueName: \"kubernetes.io/projected/365a1938-752c-4cec-a917-c5a46c719307-kube-api-access-qdv4k\") pod \"ingress-canary-vhccw\" (UID: \"365a1938-752c-4cec-a917-c5a46c719307\") " pod="openshift-ingress-canary/ingress-canary-vhccw"
Nov 03 22:12:57.214326 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.214300    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-ovnkube-script-lib\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:57.214326 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.214306    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b071df2-94c8-4459-89f2-f4b448ceb581-metrics-certs\") pod \"network-metrics-daemon-jn5wf\" (UID: \"9b071df2-94c8-4459-89f2-f4b448ceb581\") " pod="openshift-multus/network-metrics-daemon-jn5wf"
Nov 03 22:12:57.214521 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.214389    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8187068e-17ec-4e6d-bb8e-1d0312aa6674-agent-certs\") pod \"konnectivity-agent-7p8pl\" (UID: \"8187068e-17ec-4e6d-bb8e-1d0312aa6674\") " pod="kube-system/konnectivity-agent-7p8pl"
Nov 03 22:12:57.214521 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.214424    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/089f9a4b-8eec-4758-811b-e506b41cecbb-system-cni-dir\") pod \"multus-additional-cni-plugins-4sm9h\" (UID: \"089f9a4b-8eec-4758-811b-e506b41cecbb\") " pod="openshift-multus/multus-additional-cni-plugins-4sm9h"
Nov 03 22:12:57.214521 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.214468    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/089f9a4b-8eec-4758-811b-e506b41cecbb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4sm9h\" (UID: \"089f9a4b-8eec-4758-811b-e506b41cecbb\") " pod="openshift-multus/multus-additional-cni-plugins-4sm9h"
Nov 03 22:12:57.214521 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.214486    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-ovnkube-config\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:57.214521 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.214506    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bjhs\" (UniqueName: \"kubernetes.io/projected/089f9a4b-8eec-4758-811b-e506b41cecbb-kube-api-access-7bjhs\") pod \"multus-additional-cni-plugins-4sm9h\" (UID: \"089f9a4b-8eec-4758-811b-e506b41cecbb\") " pod="openshift-multus/multus-additional-cni-plugins-4sm9h"
Nov 03 22:12:57.214709 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.214612    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-host-kubelet\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:57.214709 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.214677    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-system-cni-dir\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:57.214709 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.214706    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-cni-binary-copy\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:57.214798 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.214730    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-host-var-lib-cni-multus\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:57.214798 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.214768    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8ea0b360-1d72-4d40-935e-1394fecad74c-original-pull-secret\") pod \"global-pull-secret-syncer-66m8v\" (UID: \"8ea0b360-1d72-4d40-935e-1394fecad74c\") " pod="kube-system/global-pull-secret-syncer-66m8v"
Nov 03 22:12:57.214899 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.214826    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fa1fd639-c265-4048-b9be-a583885ccb4e-device-dir\") pod \"aws-ebs-csi-driver-node-8zcr2\" (UID: \"fa1fd639-c265-4048-b9be-a583885ccb4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zcr2"
Nov 03 22:12:57.214899 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.214860    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-cni-binary-copy\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:57.214996 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.214981    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8ea0b360-1d72-4d40-935e-1394fecad74c-original-pull-secret\") pod \"global-pull-secret-syncer-66m8v\" (UID: \"8ea0b360-1d72-4d40-935e-1394fecad74c\") " pod="kube-system/global-pull-secret-syncer-66m8v"
Nov 03 22:12:57.228786 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.228767    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhr4l\" (UniqueName: \"kubernetes.io/projected/9b071df2-94c8-4459-89f2-f4b448ceb581-kube-api-access-mhr4l\") pod \"network-metrics-daemon-jn5wf\" (UID: \"9b071df2-94c8-4459-89f2-f4b448ceb581\") " pod="openshift-multus/network-metrics-daemon-jn5wf"
Nov 03 22:12:57.229460 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.229445    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8slxn\" (UniqueName: \"kubernetes.io/projected/309c2503-3fef-4048-af01-6b3b70aa52a1-kube-api-access-8slxn\") pod \"dns-default-4qrls\" (UID: \"309c2503-3fef-4048-af01-6b3b70aa52a1\") " pod="openshift-dns/dns-default-4qrls"
Nov 03 22:12:57.229654 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.229638    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fswpg\" (UniqueName: \"kubernetes.io/projected/94868020-2e98-4295-8db5-043461cc6e9d-kube-api-access-fswpg\") pod \"node-resolver-x7cpm\" (UID: \"94868020-2e98-4295-8db5-043461cc6e9d\") " pod="openshift-dns/node-resolver-x7cpm"
Nov 03 22:12:57.230312 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.230292    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6x5m\" (UniqueName: \"kubernetes.io/projected/867a2a76-8aa3-4b07-98bf-913c2d2a5562-kube-api-access-f6x5m\") pod \"iptables-alerter-7rqgq\" (UID: \"867a2a76-8aa3-4b07-98bf-913c2d2a5562\") " pod="openshift-network-operator/iptables-alerter-7rqgq"
Nov 03 22:12:57.233127 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.233110    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7l82\" (UniqueName: \"kubernetes.io/projected/a06ec3e9-0a6e-46d8-ade5-13d657a6284c-kube-api-access-t7l82\") pod \"tuned-9ltv9\" (UID: \"a06ec3e9-0a6e-46d8-ade5-13d657a6284c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ltv9"
Nov 03 22:12:57.233578 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.233562    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgx8f\" (UniqueName: \"kubernetes.io/projected/6498eed3-6528-4a49-a5d6-3dfcf9d75017-kube-api-access-xgx8f\") pod \"migrator-66df5d496b-8dbwq\" (UID: \"6498eed3-6528-4a49-a5d6-3dfcf9d75017\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8dbwq"
Nov 03 22:12:57.234247 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.234232    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrr2v\" (UniqueName: \"kubernetes.io/projected/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-kube-api-access-nrr2v\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:57.234407 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.234356    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shrkm\" (UniqueName: \"kubernetes.io/projected/fa1fd639-c265-4048-b9be-a583885ccb4e-kube-api-access-shrkm\") pod \"aws-ebs-csi-driver-node-8zcr2\" (UID: \"fa1fd639-c265-4048-b9be-a583885ccb4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zcr2"
Nov 03 22:12:57.234986 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.234963    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6l77\" (UniqueName: \"kubernetes.io/projected/6813e008-7728-4466-979d-3ba72892a647-kube-api-access-b6l77\") pod \"insights-runtime-extractor-q6ndx\" (UID: \"6813e008-7728-4466-979d-3ba72892a647\") " pod="openshift-insights/insights-runtime-extractor-q6ndx"
Nov 03 22:12:57.235061 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.235045    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-png47\" (UniqueName: \"kubernetes.io/projected/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-kube-api-access-png47\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:57.235181 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.235161    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ssqq\" (UniqueName: \"kubernetes.io/projected/cc66a3de-7e7e-4855-8f17-db97dcfa815a-kube-api-access-2ssqq\") pod \"node-ca-p4p25\" (UID: \"cc66a3de-7e7e-4855-8f17-db97dcfa815a\") " pod="openshift-image-registry/node-ca-p4p25"
Nov 03 22:12:57.235258 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.235178    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bjhs\" (UniqueName: \"kubernetes.io/projected/089f9a4b-8eec-4758-811b-e506b41cecbb-kube-api-access-7bjhs\") pod \"multus-additional-cni-plugins-4sm9h\" (UID: \"089f9a4b-8eec-4758-811b-e506b41cecbb\") " pod="openshift-multus/multus-additional-cni-plugins-4sm9h"
Nov 03 22:12:57.235548 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.235529    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdv4k\" (UniqueName: \"kubernetes.io/projected/365a1938-752c-4cec-a917-c5a46c719307-kube-api-access-qdv4k\") pod \"ingress-canary-vhccw\" (UID: \"365a1938-752c-4cec-a917-c5a46c719307\") " pod="openshift-ingress-canary/ingress-canary-vhccw"
Nov 03 22:12:57.235644 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.235629    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdtpv\" (UniqueName: \"kubernetes.io/projected/1b792dc0-5e1c-40c4-a6c6-f68b4c6c8aac-kube-api-access-qdtpv\") pod \"network-check-target-82h4r\" (UID: \"1b792dc0-5e1c-40c4-a6c6-f68b4c6c8aac\") " pod="openshift-network-diagnostics/network-check-target-82h4r"
Nov 03 22:12:57.315395 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.315318    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a06ec3e9-0a6e-46d8-ade5-13d657a6284c-host\") pod \"tuned-9ltv9\" (UID: \"a06ec3e9-0a6e-46d8-ade5-13d657a6284c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ltv9"
Nov 03 22:12:57.315395 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.315349    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a06ec3e9-0a6e-46d8-ade5-13d657a6284c-etc-kubernetes\") pod \"tuned-9ltv9\" (UID: \"a06ec3e9-0a6e-46d8-ade5-13d657a6284c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ltv9"
Nov 03 22:12:57.315395 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.315368    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a06ec3e9-0a6e-46d8-ade5-13d657a6284c-var-lib-kubelet\") pod \"tuned-9ltv9\" (UID: \"a06ec3e9-0a6e-46d8-ade5-13d657a6284c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ltv9"
Nov 03 22:12:57.315395 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.315386    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-host-slash\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:57.315664 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.315410    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a06ec3e9-0a6e-46d8-ade5-13d657a6284c-etc-sysconfig\") pod \"tuned-9ltv9\" (UID: \"a06ec3e9-0a6e-46d8-ade5-13d657a6284c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ltv9"
Nov 03 22:12:57.315664 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.315431    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-cnibin\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:57.315664 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.315455    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/867a2a76-8aa3-4b07-98bf-913c2d2a5562-host-slash\") pod \"iptables-alerter-7rqgq\" (UID: \"867a2a76-8aa3-4b07-98bf-913c2d2a5562\") " pod="openshift-network-operator/iptables-alerter-7rqgq"
Nov 03 22:12:57.315664 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.315458    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a06ec3e9-0a6e-46d8-ade5-13d657a6284c-etc-kubernetes\") pod \"tuned-9ltv9\" (UID: \"a06ec3e9-0a6e-46d8-ade5-13d657a6284c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ltv9"
Nov 03 22:12:57.315664 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.315470    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a06ec3e9-0a6e-46d8-ade5-13d657a6284c-var-lib-kubelet\") pod \"tuned-9ltv9\" (UID: \"a06ec3e9-0a6e-46d8-ade5-13d657a6284c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ltv9"
Nov 03 22:12:57.315664 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.315479    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-host-slash\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:57.315664 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.315501    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8ea0b360-1d72-4d40-935e-1394fecad74c-dbus\") pod \"global-pull-secret-syncer-66m8v\" (UID: \"8ea0b360-1d72-4d40-935e-1394fecad74c\") " pod="kube-system/global-pull-secret-syncer-66m8v"
Nov 03 22:12:57.315664 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.315523    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-cnibin\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:57.315664 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.315533    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/867a2a76-8aa3-4b07-98bf-913c2d2a5562-host-slash\") pod \"iptables-alerter-7rqgq\" (UID: \"867a2a76-8aa3-4b07-98bf-913c2d2a5562\") " pod="openshift-network-operator/iptables-alerter-7rqgq"
Nov 03 22:12:57.315664 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.315554    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8ea0b360-1d72-4d40-935e-1394fecad74c-dbus\") pod \"global-pull-secret-syncer-66m8v\" (UID: \"8ea0b360-1d72-4d40-935e-1394fecad74c\") " pod="kube-system/global-pull-secret-syncer-66m8v"
Nov 03 22:12:57.315664 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.315525    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-var-lib-openvswitch\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:57.315664 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.315450    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a06ec3e9-0a6e-46d8-ade5-13d657a6284c-host\") pod \"tuned-9ltv9\" (UID: \"a06ec3e9-0a6e-46d8-ade5-13d657a6284c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ltv9"
Nov 03 22:12:57.315664 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.315564    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-var-lib-openvswitch\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:57.315664 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.315591    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a06ec3e9-0a6e-46d8-ade5-13d657a6284c-run\") pod \"tuned-9ltv9\" (UID: \"a06ec3e9-0a6e-46d8-ade5-13d657a6284c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ltv9"
Nov 03 22:12:57.315664 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.315609    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-host-run-k8s-cni-cncf-io\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:57.315664 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.315499    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a06ec3e9-0a6e-46d8-ade5-13d657a6284c-etc-sysconfig\") pod \"tuned-9ltv9\" (UID: \"a06ec3e9-0a6e-46d8-ade5-13d657a6284c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ltv9"
Nov 03 22:12:57.315664 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.315627    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-etc-openvswitch\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:57.315664 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.315649    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-host-run-ovn-kubernetes\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:57.315664 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.315664    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a06ec3e9-0a6e-46d8-ade5-13d657a6284c-run\") pod \"tuned-9ltv9\" (UID: \"a06ec3e9-0a6e-46d8-ade5-13d657a6284c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ltv9"
Nov 03 22:12:57.315664 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.315670    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fa1fd639-c265-4048-b9be-a583885ccb4e-registration-dir\") pod \"aws-ebs-csi-driver-node-8zcr2\" (UID: \"fa1fd639-c265-4048-b9be-a583885ccb4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zcr2"
Nov 03 22:12:57.316530 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.315679    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-host-run-k8s-cni-cncf-io\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:57.316530 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.315684    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-etc-openvswitch\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:57.316530 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.315695    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-host-run-ovn-kubernetes\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:57.316530 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.315687    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b071df2-94c8-4459-89f2-f4b448ceb581-metrics-certs\") pod \"network-metrics-daemon-jn5wf\" (UID: \"9b071df2-94c8-4459-89f2-f4b448ceb581\") " pod="openshift-multus/network-metrics-daemon-jn5wf"
Nov 03 22:12:57.316530 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.315730    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/089f9a4b-8eec-4758-811b-e506b41cecbb-system-cni-dir\") pod \"multus-additional-cni-plugins-4sm9h\" (UID: \"089f9a4b-8eec-4758-811b-e506b41cecbb\") " pod="openshift-multus/multus-additional-cni-plugins-4sm9h"
Nov 03 22:12:57.316530 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.315745    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/089f9a4b-8eec-4758-811b-e506b41cecbb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4sm9h\" (UID: \"089f9a4b-8eec-4758-811b-e506b41cecbb\") " pod="openshift-multus/multus-additional-cni-plugins-4sm9h"
Nov 03 22:12:57.316530 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.315750    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fa1fd639-c265-4048-b9be-a583885ccb4e-registration-dir\") pod \"aws-ebs-csi-driver-node-8zcr2\" (UID: \"fa1fd639-c265-4048-b9be-a583885ccb4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zcr2"
Nov 03 22:12:57.316530 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.315765    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/089f9a4b-8eec-4758-811b-e506b41cecbb-system-cni-dir\") pod \"multus-additional-cni-plugins-4sm9h\" (UID: \"089f9a4b-8eec-4758-811b-e506b41cecbb\") " pod="openshift-multus/multus-additional-cni-plugins-4sm9h"
Nov 03 22:12:57.316530 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.315760    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-host-kubelet\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:57.316530 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.315790    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-host-kubelet\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:57.316530 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.315793    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-system-cni-dir\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:57.316530 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.315807    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/089f9a4b-8eec-4758-811b-e506b41cecbb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4sm9h\" (UID: \"089f9a4b-8eec-4758-811b-e506b41cecbb\") " pod="openshift-multus/multus-additional-cni-plugins-4sm9h"
Nov 03 22:12:57.316530 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.315844    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-system-cni-dir\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:57.316530 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.315843    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-host-var-lib-cni-multus\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:57.316530 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.315875    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-host-var-lib-cni-multus\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:57.316530 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.315879    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fa1fd639-c265-4048-b9be-a583885ccb4e-device-dir\") pod \"aws-ebs-csi-driver-node-8zcr2\" (UID: \"fa1fd639-c265-4048-b9be-a583885ccb4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zcr2"
Nov 03 22:12:57.316530 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.315913    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-systemd-units\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:57.316530 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.315935    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-systemd-units\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:57.316530 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.315938    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a06ec3e9-0a6e-46d8-ade5-13d657a6284c-etc-sysctl-conf\") pod \"tuned-9ltv9\" (UID: \"a06ec3e9-0a6e-46d8-ade5-13d657a6284c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ltv9"
Nov 03 22:12:57.316530 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.315981    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a06ec3e9-0a6e-46d8-ade5-13d657a6284c-lib-modules\") pod \"tuned-9ltv9\" (UID: \"a06ec3e9-0a6e-46d8-ade5-13d657a6284c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ltv9"
Nov 03 22:12:57.316530 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.315914    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fa1fd639-c265-4048-b9be-a583885ccb4e-device-dir\") pod \"aws-ebs-csi-driver-node-8zcr2\" (UID: \"fa1fd639-c265-4048-b9be-a583885ccb4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zcr2"
Nov 03 22:12:57.316530 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.316006    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/089f9a4b-8eec-4758-811b-e506b41cecbb-cnibin\") pod \"multus-additional-cni-plugins-4sm9h\" (UID: \"089f9a4b-8eec-4758-811b-e506b41cecbb\") " pod="openshift-multus/multus-additional-cni-plugins-4sm9h"
Nov 03 22:12:57.316530 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.316030    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-run-ovn\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:57.316530 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.316045    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a06ec3e9-0a6e-46d8-ade5-13d657a6284c-etc-sysctl-conf\") pod \"tuned-9ltv9\" (UID: \"a06ec3e9-0a6e-46d8-ade5-13d657a6284c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ltv9"
Nov 03 22:12:57.316530 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.316027    4431 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 22:12:57.316530 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.316070    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-log-socket\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:57.316530 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.316076    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/089f9a4b-8eec-4758-811b-e506b41cecbb-cnibin\") pod \"multus-additional-cni-plugins-4sm9h\" (UID: \"089f9a4b-8eec-4758-811b-e506b41cecbb\") " pod="openshift-multus/multus-additional-cni-plugins-4sm9h"
Nov 03 22:12:57.316530 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.316084    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a06ec3e9-0a6e-46d8-ade5-13d657a6284c-lib-modules\") pod \"tuned-9ltv9\" (UID: \"a06ec3e9-0a6e-46d8-ade5-13d657a6284c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ltv9"
Nov 03 22:12:57.316530 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.316098    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a06ec3e9-0a6e-46d8-ade5-13d657a6284c-sys\") pod \"tuned-9ltv9\" (UID: \"a06ec3e9-0a6e-46d8-ade5-13d657a6284c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ltv9"
Nov 03 22:12:57.316530 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.316109    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-log-socket\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:57.316530 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.316098    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-run-ovn\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:57.316530 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.316126    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8ea0b360-1d72-4d40-935e-1394fecad74c-kubelet-config\") pod \"global-pull-secret-syncer-66m8v\" (UID: \"8ea0b360-1d72-4d40-935e-1394fecad74c\") " pod="kube-system/global-pull-secret-syncer-66m8v"
Nov 03 22:12:57.316530 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.316150    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fa1fd639-c265-4048-b9be-a583885ccb4e-socket-dir\") pod \"aws-ebs-csi-driver-node-8zcr2\" (UID: \"fa1fd639-c265-4048-b9be-a583885ccb4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zcr2"
Nov 03 22:12:57.316530 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.316137    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a06ec3e9-0a6e-46d8-ade5-13d657a6284c-sys\") pod \"tuned-9ltv9\" (UID: \"a06ec3e9-0a6e-46d8-ade5-13d657a6284c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ltv9"
Nov 03 22:12:57.316530 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.316159    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8ea0b360-1d72-4d40-935e-1394fecad74c-kubelet-config\") pod \"global-pull-secret-syncer-66m8v\" (UID: \"8ea0b360-1d72-4d40-935e-1394fecad74c\") " pod="kube-system/global-pull-secret-syncer-66m8v"
Nov 03 22:12:57.316530 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.316176    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-multus-cni-dir\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:57.316530 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.316202    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-host-var-lib-cni-bin\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:57.316530 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.316240    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fa1fd639-c265-4048-b9be-a583885ccb4e-socket-dir\") pod \"aws-ebs-csi-driver-node-8zcr2\" (UID: \"fa1fd639-c265-4048-b9be-a583885ccb4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zcr2"
Nov 03 22:12:57.316530 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.316246    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-multus-conf-dir\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:57.316530 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.316269    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-host-var-lib-cni-bin\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:57.316530 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.316272    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-multus-cni-dir\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:57.316530 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.316271    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/309c2503-3fef-4048-af01-6b3b70aa52a1-metrics-tls\") pod \"dns-default-4qrls\" (UID: \"309c2503-3fef-4048-af01-6b3b70aa52a1\") " pod="openshift-dns/dns-default-4qrls"
Nov 03 22:12:57.316530 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.316295    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-multus-conf-dir\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:57.316530 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.316308    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-run-openvswitch\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:57.316530 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.316327    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-node-log\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:57.316530 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.316343    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-multus-socket-dir-parent\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:57.316530 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.316363    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cc66a3de-7e7e-4855-8f17-db97dcfa815a-host\") pod \"node-ca-p4p25\" (UID: \"cc66a3de-7e7e-4855-8f17-db97dcfa815a\") " pod="openshift-image-registry/node-ca-p4p25"
Nov 03 22:12:57.316530 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.316374    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-run-openvswitch\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:57.316530 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.316401    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-node-log\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:57.316530 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.316404    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-host-run-netns\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:57.316530 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.316421    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-multus-socket-dir-parent\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:57.316530 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:12:57.316361    4431 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 22:12:57.316530 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.316433    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fa1fd639-c265-4048-b9be-a583885ccb4e-sys-fs\") pod \"aws-ebs-csi-driver-node-8zcr2\" (UID: \"fa1fd639-c265-4048-b9be-a583885ccb4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zcr2"
Nov 03 22:12:57.316530 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.316440    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cc66a3de-7e7e-4855-8f17-db97dcfa815a-host\") pod \"node-ca-p4p25\" (UID: \"cc66a3de-7e7e-4855-8f17-db97dcfa815a\") " pod="openshift-image-registry/node-ca-p4p25"
Nov 03 22:12:57.316530 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.316460    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fa1fd639-c265-4048-b9be-a583885ccb4e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8zcr2\" (UID: \"fa1fd639-c265-4048-b9be-a583885ccb4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zcr2"
Nov 03 22:12:57.316530 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.316482    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fa1fd639-c265-4048-b9be-a583885ccb4e-sys-fs\") pod \"aws-ebs-csi-driver-node-8zcr2\" (UID: \"fa1fd639-c265-4048-b9be-a583885ccb4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zcr2"
Nov 03 22:12:57.316530 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.316488    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fa1fd639-c265-4048-b9be-a583885ccb4e-etc-selinux\") pod \"aws-ebs-csi-driver-node-8zcr2\" (UID: \"fa1fd639-c265-4048-b9be-a583885ccb4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zcr2"
Nov 03 22:12:57.316530 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.316495    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-host-run-netns\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:57.316530 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:12:57.316505    4431 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/309c2503-3fef-4048-af01-6b3b70aa52a1-metrics-tls podName:309c2503-3fef-4048-af01-6b3b70aa52a1 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:57.816489911 +0000 UTC m=+1.842345751 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/309c2503-3fef-4048-af01-6b3b70aa52a1-metrics-tls") pod "dns-default-4qrls" (UID: "309c2503-3fef-4048-af01-6b3b70aa52a1") : secret "dns-default-metrics-tls" not found
Nov 03 22:12:57.316530 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.316523    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fa1fd639-c265-4048-b9be-a583885ccb4e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8zcr2\" (UID: \"fa1fd639-c265-4048-b9be-a583885ccb4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zcr2"
Nov 03 22:12:57.316530 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.316540    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-host-var-lib-kubelet\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:57.316530 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.316565    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6813e008-7728-4466-979d-3ba72892a647-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q6ndx\" (UID: \"6813e008-7728-4466-979d-3ba72892a647\") " pod="openshift-insights/insights-runtime-extractor-q6ndx"
Nov 03 22:12:57.316530 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.316571    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-host-var-lib-kubelet\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:57.318688 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.316542    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fa1fd639-c265-4048-b9be-a583885ccb4e-etc-selinux\") pod \"aws-ebs-csi-driver-node-8zcr2\" (UID: \"fa1fd639-c265-4048-b9be-a583885ccb4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zcr2"
Nov 03 22:12:57.318688 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.316588    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/94868020-2e98-4295-8db5-043461cc6e9d-hosts-file\") pod \"node-resolver-x7cpm\" (UID: \"94868020-2e98-4295-8db5-043461cc6e9d\") " pod="openshift-dns/node-resolver-x7cpm"
Nov 03 22:12:57.318688 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.316611    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-host-run-netns\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:57.318688 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.316634    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-host-cni-bin\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:57.318688 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.316658    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-host-run-netns\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:57.318688 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.316658    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-host-cni-netd\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:57.318688 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.316636    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/94868020-2e98-4295-8db5-043461cc6e9d-hosts-file\") pod \"node-resolver-x7cpm\" (UID: \"94868020-2e98-4295-8db5-043461cc6e9d\") " pod="openshift-dns/node-resolver-x7cpm"
Nov 03 22:12:57.318688 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.316694    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-host-cni-bin\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:57.318688 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.316703    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-host-cni-netd\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:57.318688 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.316750    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-host-run-multus-certs\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:57.318688 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.316775    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6813e008-7728-4466-979d-3ba72892a647-crio-socket\") pod \"insights-runtime-extractor-q6ndx\" (UID: \"6813e008-7728-4466-979d-3ba72892a647\") " pod="openshift-insights/insights-runtime-extractor-q6ndx"
Nov 03 22:12:57.318688 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.316802    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-run-systemd\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:57.318688 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.316836    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a06ec3e9-0a6e-46d8-ade5-13d657a6284c-etc-modprobe-d\") pod \"tuned-9ltv9\" (UID: \"a06ec3e9-0a6e-46d8-ade5-13d657a6284c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ltv9"
Nov 03 22:12:57.318688 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.316838    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-host-run-multus-certs\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:57.318688 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.316856    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6813e008-7728-4466-979d-3ba72892a647-crio-socket\") pod \"insights-runtime-extractor-q6ndx\" (UID: \"6813e008-7728-4466-979d-3ba72892a647\") " pod="openshift-insights/insights-runtime-extractor-q6ndx"
Nov 03 22:12:57.318688 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.316841    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-run-systemd\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:57.318688 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.316873    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a06ec3e9-0a6e-46d8-ade5-13d657a6284c-etc-systemd\") pod \"tuned-9ltv9\" (UID: \"a06ec3e9-0a6e-46d8-ade5-13d657a6284c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ltv9"
Nov 03 22:12:57.318688 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.316905    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6813e008-7728-4466-979d-3ba72892a647-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q6ndx\" (UID: \"6813e008-7728-4466-979d-3ba72892a647\") " pod="openshift-insights/insights-runtime-extractor-q6ndx"
Nov 03 22:12:57.318688 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.316928    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a06ec3e9-0a6e-46d8-ade5-13d657a6284c-etc-modprobe-d\") pod \"tuned-9ltv9\" (UID: \"a06ec3e9-0a6e-46d8-ade5-13d657a6284c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ltv9"
Nov 03 22:12:57.318688 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.316952    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a06ec3e9-0a6e-46d8-ade5-13d657a6284c-etc-sysctl-d\") pod \"tuned-9ltv9\" (UID: \"a06ec3e9-0a6e-46d8-ade5-13d657a6284c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ltv9"
Nov 03 22:12:57.318688 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.316947    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a06ec3e9-0a6e-46d8-ade5-13d657a6284c-etc-systemd\") pod \"tuned-9ltv9\" (UID: \"a06ec3e9-0a6e-46d8-ade5-13d657a6284c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ltv9"
Nov 03 22:12:57.318688 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.316996    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-hostroot\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:57.318688 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.317025    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-etc-kubernetes\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:57.318688 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.317041    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-hostroot\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:57.318688 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.317056    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6813e008-7728-4466-979d-3ba72892a647-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q6ndx\" (UID: \"6813e008-7728-4466-979d-3ba72892a647\") " pod="openshift-insights/insights-runtime-extractor-q6ndx"
Nov 03 22:12:57.318688 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.317056    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:57.318688 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.317093    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac5ecdf9-3675-4d4f-98e3-a4e8796669be-etc-kubernetes\") pod \"multus-2k9fs\" (UID: \"ac5ecdf9-3675-4d4f-98e3-a4e8796669be\") " pod="openshift-multus/multus-2k9fs"
Nov 03 22:12:57.318688 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.317123    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mx5q8\" (UID: \"ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:57.318688 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.317058    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a06ec3e9-0a6e-46d8-ade5-13d657a6284c-etc-sysctl-d\") pod \"tuned-9ltv9\" (UID: \"a06ec3e9-0a6e-46d8-ade5-13d657a6284c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ltv9"
Nov 03 22:12:57.318688 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.318371    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b071df2-94c8-4459-89f2-f4b448ceb581-metrics-certs\") pod \"network-metrics-daemon-jn5wf\" (UID: \"9b071df2-94c8-4459-89f2-f4b448ceb581\") " pod="openshift-multus/network-metrics-daemon-jn5wf"
Nov 03 22:12:57.319794 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.319529    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6813e008-7728-4466-979d-3ba72892a647-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q6ndx\" (UID: \"6813e008-7728-4466-979d-3ba72892a647\") " pod="openshift-insights/insights-runtime-extractor-q6ndx"
Nov 03 22:12:57.380401 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.380371    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jn5wf"
Nov 03 22:12:57.386992 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.386973    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-q6ndx"
Nov 03 22:12:57.691021 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.690999    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vhccw_365a1938-752c-4cec-a917-c5a46c719307/serve-healthcheck-canary/0.log"
Nov 03 22:12:57.719734 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.719708    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jn5wf"]
Nov 03 22:12:57.723230 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:57.723184    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b071df2_94c8_4459_89f2_f4b448ceb581.slice/crio-1e6af5ec02176e282129b78995dea49c778085bbb53982cf6dd3de5c635f5649 WatchSource:0}: Error finding container 1e6af5ec02176e282129b78995dea49c778085bbb53982cf6dd3de5c635f5649: Status 404 returned error can't find the container with id 1e6af5ec02176e282129b78995dea49c778085bbb53982cf6dd3de5c635f5649
Nov 03 22:12:57.725296 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.725281    4431 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 22:12:57.747779 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.747747    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-q6ndx"]
Nov 03 22:12:57.751353 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:12:57.751327    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6813e008_7728_4466_979d_3ba72892a647.slice/crio-48bc710bbe1aed6a99eadd74ea780961e1279d3cacfd8a71c820919012c1cb6c WatchSource:0}: Error finding container 48bc710bbe1aed6a99eadd74ea780961e1279d3cacfd8a71c820919012c1cb6c: Status 404 returned error can't find the container with id 48bc710bbe1aed6a99eadd74ea780961e1279d3cacfd8a71c820919012c1cb6c
Nov 03 22:12:57.822141 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.822117    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/309c2503-3fef-4048-af01-6b3b70aa52a1-metrics-tls\") pod \"dns-default-4qrls\" (UID: \"309c2503-3fef-4048-af01-6b3b70aa52a1\") " pod="openshift-dns/dns-default-4qrls"
Nov 03 22:12:57.822289 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:12:57.822270    4431 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 22:12:57.822338 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:12:57.822329    4431 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/309c2503-3fef-4048-af01-6b3b70aa52a1-metrics-tls podName:309c2503-3fef-4048-af01-6b3b70aa52a1 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:58.822315411 +0000 UTC m=+2.848171257 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/309c2503-3fef-4048-af01-6b3b70aa52a1-metrics-tls") pod "dns-default-4qrls" (UID: "309c2503-3fef-4048-af01-6b3b70aa52a1") : secret "dns-default-metrics-tls" not found
Nov 03 22:12:57.837841 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.837818    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:12:57.982821 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:57.982742    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:12:58.119624 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:58.119590    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jn5wf" event={"ID":"9b071df2-94c8-4459-89f2-f4b448ceb581","Type":"ContainerStarted","Data":"1e6af5ec02176e282129b78995dea49c778085bbb53982cf6dd3de5c635f5649"}
Nov 03 22:12:58.121116 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:58.121091    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q6ndx" event={"ID":"6813e008-7728-4466-979d-3ba72892a647","Type":"ContainerStarted","Data":"5b63f4553d48e87d227bd2dcf72252aa0c46b3352573fe2540a217ababc086ee"}
Nov 03 22:12:58.121241 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:58.121134    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q6ndx" event={"ID":"6813e008-7728-4466-979d-3ba72892a647","Type":"ContainerStarted","Data":"48bc710bbe1aed6a99eadd74ea780961e1279d3cacfd8a71c820919012c1cb6c"}
Nov 03 22:12:58.289759 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:58.289677    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-8dbwq_6498eed3-6528-4a49-a5d6-3dfcf9d75017/migrator/0.log"
Nov 03 22:12:58.304770 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:58.304732    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:58.389766 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:58.389735    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:12:58.508537 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:58.508513    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:58.595056 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:58.594979    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:12:58.765463 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:58.765434    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mx5q8"
Nov 03 22:12:58.786499 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:58.786470    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:12:58.827528 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:58.827497    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/309c2503-3fef-4048-af01-6b3b70aa52a1-metrics-tls\") pod \"dns-default-4qrls\" (UID: \"309c2503-3fef-4048-af01-6b3b70aa52a1\") " pod="openshift-dns/dns-default-4qrls"
Nov 03 22:12:58.827660 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:12:58.827604    4431 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 22:12:58.827660 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:12:58.827652    4431 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/309c2503-3fef-4048-af01-6b3b70aa52a1-metrics-tls podName:309c2503-3fef-4048-af01-6b3b70aa52a1 nodeName:}" failed. No retries permitted until 2026-11-03 22:13:00.827639821 +0000 UTC m=+4.853495660 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/309c2503-3fef-4048-af01-6b3b70aa52a1-metrics-tls") pod "dns-default-4qrls" (UID: "309c2503-3fef-4048-af01-6b3b70aa52a1") : secret "dns-default-metrics-tls" not found
Nov 03 22:12:59.079352 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:59.079324    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:12:59.125283 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:59.125244    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q6ndx" event={"ID":"6813e008-7728-4466-979d-3ba72892a647","Type":"ContainerStarted","Data":"1aec1eb8e66db50b32a4b42a4fd38918132588a512060897240155ef2f9c6fe4"}
Nov 03 22:12:59.216381 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:59.216315    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:12:59.474316 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:59.474244    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:12:59.986345 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:12:59.986308    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:13:00.131153 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:00.131102    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jn5wf" event={"ID":"9b071df2-94c8-4459-89f2-f4b448ceb581","Type":"ContainerStarted","Data":"d0df29d5e538bbf8b1d64f62910eeff3cd2924dbb92d8658e19f61fa1952e68d"}
Nov 03 22:13:00.131607 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:00.131179    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jn5wf" event={"ID":"9b071df2-94c8-4459-89f2-f4b448ceb581","Type":"ContainerStarted","Data":"184f058da75dc4b0c73b4680559094f41e85d9e76d7e9d22094cc9b256d44745"}
Nov 03 22:13:00.378795 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:00.378709    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:13:00.577177 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:00.577136    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:13:00.780055 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:00.780020    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:13:00.841629 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:00.841604    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/309c2503-3fef-4048-af01-6b3b70aa52a1-metrics-tls\") pod \"dns-default-4qrls\" (UID: \"309c2503-3fef-4048-af01-6b3b70aa52a1\") " pod="openshift-dns/dns-default-4qrls"
Nov 03 22:13:00.841751 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:13:00.841737    4431 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 22:13:00.841813 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:13:00.841802    4431 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/309c2503-3fef-4048-af01-6b3b70aa52a1-metrics-tls podName:309c2503-3fef-4048-af01-6b3b70aa52a1 nodeName:}" failed. No retries permitted until 2026-11-03 22:13:04.841787742 +0000 UTC m=+8.867643582 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/309c2503-3fef-4048-af01-6b3b70aa52a1-metrics-tls") pod "dns-default-4qrls" (UID: "309c2503-3fef-4048-af01-6b3b70aa52a1") : secret "dns-default-metrics-tls" not found
Nov 03 22:13:00.981655 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:00.981625    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:13:01.136454 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:01.136366    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q6ndx" event={"ID":"6813e008-7728-4466-979d-3ba72892a647","Type":"ContainerStarted","Data":"3b79d6fdd9cba89e40097ec8bffdbd859576412add0af5274057a4353c2e3a63"}
Nov 03 22:13:01.257000 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:13:01.256966    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:13:02.184744 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:02.184712    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:13:02.322750 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:02.322705    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:13:02.462428 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:02.462336    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:13:02.605270 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:02.605238    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:13:02.808262 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:02.808161    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:13:02.951863 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:02.951832    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:13:03.087333 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:03.087256    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:13:03.939027 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:03.938994    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:13:04.173915 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:04.173884    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:13:04.375490 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:04.375404    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:13:04.581923 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:04.581880    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:13:04.788476 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:04.788438    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:13:04.864676 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:04.864647    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/309c2503-3fef-4048-af01-6b3b70aa52a1-metrics-tls\") pod \"dns-default-4qrls\" (UID: \"309c2503-3fef-4048-af01-6b3b70aa52a1\") " pod="openshift-dns/dns-default-4qrls"
Nov 03 22:13:04.866926 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:04.866894    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/309c2503-3fef-4048-af01-6b3b70aa52a1-metrics-tls\") pod \"dns-default-4qrls\" (UID: \"309c2503-3fef-4048-af01-6b3b70aa52a1\") " pod="openshift-dns/dns-default-4qrls"
Nov 03 22:13:04.886675 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:04.886649    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4qrls"
Nov 03 22:13:04.979287 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:04.979253    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:13:05.176992 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:05.176962    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4qrls"]
Nov 03 22:13:05.180209 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:13:05.180181    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod309c2503_3fef_4048_af01_6b3b70aa52a1.slice/crio-9cd227f21cc18b0805ca2f3835a873009040ef88704926f7df64ba5f711c08e0 WatchSource:0}: Error finding container 9cd227f21cc18b0805ca2f3835a873009040ef88704926f7df64ba5f711c08e0: Status 404 returned error can't find the container with id 9cd227f21cc18b0805ca2f3835a873009040ef88704926f7df64ba5f711c08e0
Nov 03 22:13:05.583603 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:05.583525    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:13:05.782971 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:05.782934    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:13:06.149404 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:06.149366    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4qrls" event={"ID":"309c2503-3fef-4048-af01-6b3b70aa52a1","Type":"ContainerStarted","Data":"9cd227f21cc18b0805ca2f3835a873009040ef88704926f7df64ba5f711c08e0"}
Nov 03 22:13:06.369454 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:06.369417    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-82h4r"
Nov 03 22:13:07.153603 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:07.153514    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4qrls" event={"ID":"309c2503-3fef-4048-af01-6b3b70aa52a1","Type":"ContainerStarted","Data":"13a2bf4fb6e46640f588fa19db9b5330913ccddffdcaec8ad90bea677bed6819"}
Nov 03 22:13:07.153993 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:07.153607    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4qrls" event={"ID":"309c2503-3fef-4048-af01-6b3b70aa52a1","Type":"ContainerStarted","Data":"dccdb4c51eaa62abc9a535c6f1fb5c96b703dbd02cc71efcd9a4f94b997ba422"}
Nov 03 22:13:07.153993 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:07.153638    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-4qrls"
Nov 03 22:13:09.444459 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:09.444417    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lkstc"]
Nov 03 22:13:09.446581 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:09.446562    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lkstc"
Nov 03 22:13:09.469822 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:09.469795    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-9p5v8\""
Nov 03 22:13:09.469950 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:09.469857    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Nov 03 22:13:09.482845 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:09.482821    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lkstc"]
Nov 03 22:13:09.535961 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:09.535919    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-gzwb4"]
Nov 03 22:13:09.539442 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:09.538883    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-gzwb4"
Nov 03 22:13:09.542751 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:09.542726    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-fc6bcdddf-9zl84"]
Nov 03 22:13:09.544725 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:09.544707    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-fc6bcdddf-9zl84"
Nov 03 22:13:09.562523 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:09.562500    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-gzwb4"]
Nov 03 22:13:09.563381 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:09.563359    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 03 22:13:09.563638 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:09.563622    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 03 22:13:09.587469 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:09.587438    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 03 22:13:09.587594 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:09.587488    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 03 22:13:09.587594 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:09.587493    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 03 22:13:09.587676 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:09.587609    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-794rw\""
Nov 03 22:13:09.587676 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:09.587670    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 03 22:13:09.587792 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:09.587781    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-dxwv6\""
Nov 03 22:13:09.597913 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:09.597886    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/cdc51322-d4e3-4345-84f5-e52653531823-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-lkstc\" (UID: \"cdc51322-d4e3-4345-84f5-e52653531823\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lkstc"
Nov 03 22:13:09.598845 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:09.598822    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-fc6bcdddf-9zl84"]
Nov 03 22:13:09.628563 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:09.628529    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:13:09.699115 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:09.699027    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1bdb4a48-dd3c-4037-8130-78c55a7121b5-installation-pull-secrets\") pod \"image-registry-fc6bcdddf-9zl84\" (UID: \"1bdb4a48-dd3c-4037-8130-78c55a7121b5\") " pod="openshift-image-registry/image-registry-fc6bcdddf-9zl84"
Nov 03 22:13:09.699115 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:09.699068    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1bdb4a48-dd3c-4037-8130-78c55a7121b5-trusted-ca\") pod \"image-registry-fc6bcdddf-9zl84\" (UID: \"1bdb4a48-dd3c-4037-8130-78c55a7121b5\") " pod="openshift-image-registry/image-registry-fc6bcdddf-9zl84"
Nov 03 22:13:09.699115 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:09.699094    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1bdb4a48-dd3c-4037-8130-78c55a7121b5-bound-sa-token\") pod \"image-registry-fc6bcdddf-9zl84\" (UID: \"1bdb4a48-dd3c-4037-8130-78c55a7121b5\") " pod="openshift-image-registry/image-registry-fc6bcdddf-9zl84"
Nov 03 22:13:09.699395 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:09.699225    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sxms\" (UniqueName: \"kubernetes.io/projected/1bdb4a48-dd3c-4037-8130-78c55a7121b5-kube-api-access-2sxms\") pod \"image-registry-fc6bcdddf-9zl84\" (UID: \"1bdb4a48-dd3c-4037-8130-78c55a7121b5\") " pod="openshift-image-registry/image-registry-fc6bcdddf-9zl84"
Nov 03 22:13:09.699395 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:09.699269    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1bdb4a48-dd3c-4037-8130-78c55a7121b5-image-registry-private-configuration\") pod \"image-registry-fc6bcdddf-9zl84\" (UID: \"1bdb4a48-dd3c-4037-8130-78c55a7121b5\") " pod="openshift-image-registry/image-registry-fc6bcdddf-9zl84"
Nov 03 22:13:09.699395 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:09.699306    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/cdc51322-d4e3-4345-84f5-e52653531823-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-lkstc\" (UID: \"cdc51322-d4e3-4345-84f5-e52653531823\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lkstc"
Nov 03 22:13:09.699395 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:09.699327    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrfb8\" (UniqueName: \"kubernetes.io/projected/623e390b-6a39-48cd-9d7c-4a5890c11c6e-kube-api-access-jrfb8\") pod \"downloads-6dd789c8b7-gzwb4\" (UID: \"623e390b-6a39-48cd-9d7c-4a5890c11c6e\") " pod="openshift-console/downloads-6dd789c8b7-gzwb4"
Nov 03 22:13:09.699541 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:09.699409    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1bdb4a48-dd3c-4037-8130-78c55a7121b5-registry-certificates\") pod \"image-registry-fc6bcdddf-9zl84\" (UID: \"1bdb4a48-dd3c-4037-8130-78c55a7121b5\") " pod="openshift-image-registry/image-registry-fc6bcdddf-9zl84"
Nov 03 22:13:09.699541 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:09.699444    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1bdb4a48-dd3c-4037-8130-78c55a7121b5-registry-tls\") pod \"image-registry-fc6bcdddf-9zl84\" (UID: \"1bdb4a48-dd3c-4037-8130-78c55a7121b5\") " pod="openshift-image-registry/image-registry-fc6bcdddf-9zl84"
Nov 03 22:13:09.699541 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:09.699472    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1bdb4a48-dd3c-4037-8130-78c55a7121b5-ca-trust-extracted\") pod \"image-registry-fc6bcdddf-9zl84\" (UID: \"1bdb4a48-dd3c-4037-8130-78c55a7121b5\") " pod="openshift-image-registry/image-registry-fc6bcdddf-9zl84"
Nov 03 22:13:09.699541 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:09.699530    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/623e390b-6a39-48cd-9d7c-4a5890c11c6e-tmp\") pod \"downloads-6dd789c8b7-gzwb4\" (UID: \"623e390b-6a39-48cd-9d7c-4a5890c11c6e\") " pod="openshift-console/downloads-6dd789c8b7-gzwb4"
Nov 03 22:13:09.701825 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:09.701803    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/cdc51322-d4e3-4345-84f5-e52653531823-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-lkstc\" (UID: \"cdc51322-d4e3-4345-84f5-e52653531823\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lkstc"
Nov 03 22:13:09.756718 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:09.756688    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lkstc"
Nov 03 22:13:09.772835 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:09.772807    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:13:09.800173 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:09.800138    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1bdb4a48-dd3c-4037-8130-78c55a7121b5-registry-certificates\") pod \"image-registry-fc6bcdddf-9zl84\" (UID: \"1bdb4a48-dd3c-4037-8130-78c55a7121b5\") " pod="openshift-image-registry/image-registry-fc6bcdddf-9zl84"
Nov 03 22:13:09.800355 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:09.800190    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1bdb4a48-dd3c-4037-8130-78c55a7121b5-registry-tls\") pod \"image-registry-fc6bcdddf-9zl84\" (UID: \"1bdb4a48-dd3c-4037-8130-78c55a7121b5\") " pod="openshift-image-registry/image-registry-fc6bcdddf-9zl84"
Nov 03 22:13:09.800355 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:09.800235    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1bdb4a48-dd3c-4037-8130-78c55a7121b5-ca-trust-extracted\") pod \"image-registry-fc6bcdddf-9zl84\" (UID: \"1bdb4a48-dd3c-4037-8130-78c55a7121b5\") " pod="openshift-image-registry/image-registry-fc6bcdddf-9zl84"
Nov 03 22:13:09.800355 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:09.800256    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/623e390b-6a39-48cd-9d7c-4a5890c11c6e-tmp\") pod \"downloads-6dd789c8b7-gzwb4\" (UID: \"623e390b-6a39-48cd-9d7c-4a5890c11c6e\") " pod="openshift-console/downloads-6dd789c8b7-gzwb4"
Nov 03 22:13:09.800355 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:09.800296    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1bdb4a48-dd3c-4037-8130-78c55a7121b5-installation-pull-secrets\") pod \"image-registry-fc6bcdddf-9zl84\" (UID: \"1bdb4a48-dd3c-4037-8130-78c55a7121b5\") " pod="openshift-image-registry/image-registry-fc6bcdddf-9zl84"
Nov 03 22:13:09.800355 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:09.800341    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1bdb4a48-dd3c-4037-8130-78c55a7121b5-trusted-ca\") pod \"image-registry-fc6bcdddf-9zl84\" (UID: \"1bdb4a48-dd3c-4037-8130-78c55a7121b5\") " pod="openshift-image-registry/image-registry-fc6bcdddf-9zl84"
Nov 03 22:13:09.800625 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:09.800364    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1bdb4a48-dd3c-4037-8130-78c55a7121b5-bound-sa-token\") pod \"image-registry-fc6bcdddf-9zl84\" (UID: \"1bdb4a48-dd3c-4037-8130-78c55a7121b5\") " pod="openshift-image-registry/image-registry-fc6bcdddf-9zl84"
Nov 03 22:13:09.800625 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:09.800391    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sxms\" (UniqueName: \"kubernetes.io/projected/1bdb4a48-dd3c-4037-8130-78c55a7121b5-kube-api-access-2sxms\") pod \"image-registry-fc6bcdddf-9zl84\" (UID: \"1bdb4a48-dd3c-4037-8130-78c55a7121b5\") " pod="openshift-image-registry/image-registry-fc6bcdddf-9zl84"
Nov 03 22:13:09.800625 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:09.800422    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1bdb4a48-dd3c-4037-8130-78c55a7121b5-image-registry-private-configuration\") pod \"image-registry-fc6bcdddf-9zl84\" (UID: \"1bdb4a48-dd3c-4037-8130-78c55a7121b5\") " pod="openshift-image-registry/image-registry-fc6bcdddf-9zl84"
Nov 03 22:13:09.800825 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:09.800796    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1bdb4a48-dd3c-4037-8130-78c55a7121b5-ca-trust-extracted\") pod \"image-registry-fc6bcdddf-9zl84\" (UID: \"1bdb4a48-dd3c-4037-8130-78c55a7121b5\") " pod="openshift-image-registry/image-registry-fc6bcdddf-9zl84"
Nov 03 22:13:09.800894 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:09.800865    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrfb8\" (UniqueName: \"kubernetes.io/projected/623e390b-6a39-48cd-9d7c-4a5890c11c6e-kube-api-access-jrfb8\") pod \"downloads-6dd789c8b7-gzwb4\" (UID: \"623e390b-6a39-48cd-9d7c-4a5890c11c6e\") " pod="openshift-console/downloads-6dd789c8b7-gzwb4"
Nov 03 22:13:09.801062 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:09.801041    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1bdb4a48-dd3c-4037-8130-78c55a7121b5-registry-certificates\") pod \"image-registry-fc6bcdddf-9zl84\" (UID: \"1bdb4a48-dd3c-4037-8130-78c55a7121b5\") " pod="openshift-image-registry/image-registry-fc6bcdddf-9zl84"
Nov 03 22:13:09.802147 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:09.802116    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1bdb4a48-dd3c-4037-8130-78c55a7121b5-trusted-ca\") pod \"image-registry-fc6bcdddf-9zl84\" (UID: \"1bdb4a48-dd3c-4037-8130-78c55a7121b5\") " pod="openshift-image-registry/image-registry-fc6bcdddf-9zl84"
Nov 03 22:13:09.802296 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:09.802272    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/623e390b-6a39-48cd-9d7c-4a5890c11c6e-tmp\") pod \"downloads-6dd789c8b7-gzwb4\" (UID: \"623e390b-6a39-48cd-9d7c-4a5890c11c6e\") " pod="openshift-console/downloads-6dd789c8b7-gzwb4"
Nov 03 22:13:09.804554 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:09.804509    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1bdb4a48-dd3c-4037-8130-78c55a7121b5-image-registry-private-configuration\") pod \"image-registry-fc6bcdddf-9zl84\" (UID: \"1bdb4a48-dd3c-4037-8130-78c55a7121b5\") " pod="openshift-image-registry/image-registry-fc6bcdddf-9zl84"
Nov 03 22:13:09.806541 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:09.806184    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1bdb4a48-dd3c-4037-8130-78c55a7121b5-installation-pull-secrets\") pod \"image-registry-fc6bcdddf-9zl84\" (UID: \"1bdb4a48-dd3c-4037-8130-78c55a7121b5\") " pod="openshift-image-registry/image-registry-fc6bcdddf-9zl84"
Nov 03 22:13:09.806541 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:09.806286    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1bdb4a48-dd3c-4037-8130-78c55a7121b5-registry-tls\") pod \"image-registry-fc6bcdddf-9zl84\" (UID: \"1bdb4a48-dd3c-4037-8130-78c55a7121b5\") " pod="openshift-image-registry/image-registry-fc6bcdddf-9zl84"
Nov 03 22:13:09.840144 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:09.840118    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sxms\" (UniqueName: \"kubernetes.io/projected/1bdb4a48-dd3c-4037-8130-78c55a7121b5-kube-api-access-2sxms\") pod \"image-registry-fc6bcdddf-9zl84\" (UID: \"1bdb4a48-dd3c-4037-8130-78c55a7121b5\") " pod="openshift-image-registry/image-registry-fc6bcdddf-9zl84"
Nov 03 22:13:09.840298 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:09.840124    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrfb8\" (UniqueName: \"kubernetes.io/projected/623e390b-6a39-48cd-9d7c-4a5890c11c6e-kube-api-access-jrfb8\") pod \"downloads-6dd789c8b7-gzwb4\" (UID: \"623e390b-6a39-48cd-9d7c-4a5890c11c6e\") " pod="openshift-console/downloads-6dd789c8b7-gzwb4"
Nov 03 22:13:09.847053 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:09.847027    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1bdb4a48-dd3c-4037-8130-78c55a7121b5-bound-sa-token\") pod \"image-registry-fc6bcdddf-9zl84\" (UID: \"1bdb4a48-dd3c-4037-8130-78c55a7121b5\") " pod="openshift-image-registry/image-registry-fc6bcdddf-9zl84"
Nov 03 22:13:09.852983 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:09.852968    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-gzwb4"
Nov 03 22:13:09.857383 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:09.857366    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-fc6bcdddf-9zl84"
Nov 03 22:13:09.953246 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:09.953144    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:13:10.048601 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:10.048567    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lkstc"]
Nov 03 22:13:10.051780 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:13:10.051749    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdc51322_d4e3_4345_84f5_e52653531823.slice/crio-1153e87b0b8b7a0880f103056754590946198e2b7240579a143f887d2b860fa4 WatchSource:0}: Error finding container 1153e87b0b8b7a0880f103056754590946198e2b7240579a143f887d2b860fa4: Status 404 returned error can't find the container with id 1153e87b0b8b7a0880f103056754590946198e2b7240579a143f887d2b860fa4
Nov 03 22:13:10.156765 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:10.156658    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-fc6bcdddf-9zl84"]
Nov 03 22:13:10.159149 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:13:10.159117    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bdb4a48_dd3c_4037_8130_78c55a7121b5.slice/crio-4ca71c77c2c12e6f0d00cec97850945d9246ed5824c947bd2422c26aeabbfa27 WatchSource:0}: Error finding container 4ca71c77c2c12e6f0d00cec97850945d9246ed5824c947bd2422c26aeabbfa27: Status 404 returned error can't find the container with id 4ca71c77c2c12e6f0d00cec97850945d9246ed5824c947bd2422c26aeabbfa27
Nov 03 22:13:10.163710 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:10.163684    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lkstc" event={"ID":"cdc51322-d4e3-4345-84f5-e52653531823","Type":"ContainerStarted","Data":"1153e87b0b8b7a0880f103056754590946198e2b7240579a143f887d2b860fa4"}
Nov 03 22:13:10.165445 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:10.165415    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-fc6bcdddf-9zl84" event={"ID":"1bdb4a48-dd3c-4037-8130-78c55a7121b5","Type":"ContainerStarted","Data":"4ca71c77c2c12e6f0d00cec97850945d9246ed5824c947bd2422c26aeabbfa27"}
Nov 03 22:13:10.165535 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:10.165465    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-gzwb4"]
Nov 03 22:13:10.168317 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:13:10.168292    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod623e390b_6a39_48cd_9d7c_4a5890c11c6e.slice/crio-475c89a7d29634f6049281255f8d7b1210028b56d401365544faca281fe20d2e WatchSource:0}: Error finding container 475c89a7d29634f6049281255f8d7b1210028b56d401365544faca281fe20d2e: Status 404 returned error can't find the container with id 475c89a7d29634f6049281255f8d7b1210028b56d401365544faca281fe20d2e
Nov 03 22:13:11.170263 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:11.170208    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-fc6bcdddf-9zl84" event={"ID":"1bdb4a48-dd3c-4037-8130-78c55a7121b5","Type":"ContainerStarted","Data":"0a429296073786773d363e1830e369f8354291f92f8063022aba5f85aa55113a"}
Nov 03 22:13:11.170731 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:11.170329    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-fc6bcdddf-9zl84"
Nov 03 22:13:11.172062 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:11.172033    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lkstc" event={"ID":"cdc51322-d4e3-4345-84f5-e52653531823","Type":"ContainerStarted","Data":"d03ae801ad78dea100245b0349c23f6bc8d107debe84c0631107019d8a99735b"}
Nov 03 22:13:11.172236 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:11.172200    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lkstc"
Nov 03 22:13:11.173636 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:11.173608    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-gzwb4" event={"ID":"623e390b-6a39-48cd-9d7c-4a5890c11c6e","Type":"ContainerStarted","Data":"475c89a7d29634f6049281255f8d7b1210028b56d401365544faca281fe20d2e"}
Nov 03 22:13:11.177776 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:11.177754    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lkstc"
Nov 03 22:13:11.216072 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:11.216023    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-fc6bcdddf-9zl84" podStartSLOduration=2.216007134 podStartE2EDuration="2.216007134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:13:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:13:11.215490645 +0000 UTC m=+15.241346508" watchObservedRunningTime="2026-11-03 22:13:11.216007134 +0000 UTC m=+15.241862996"
Nov 03 22:13:11.255967 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:11.255911    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lkstc" podStartSLOduration=1.4111094 podStartE2EDuration="2.255892589s" totalImagesPullingTime="844.783189ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:13:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:13:11.255050125 +0000 UTC m=+15.280905987" watchObservedRunningTime="2026-11-03 22:13:11.255892589 +0000 UTC m=+15.281748452"
Nov 03 22:13:11.574570 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:11.574475    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:13:11.652948 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:11.652910    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-f7ksv"]
Nov 03 22:13:11.655394 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:11.655372    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-f7ksv"
Nov 03 22:13:11.683395 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:11.683342    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 22:13:11.685177 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:11.683907    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 22:13:11.685177 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:11.684088    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 03 22:13:11.685177 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:11.685111    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-f7ksv"]
Nov 03 22:13:11.687034 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:11.687006    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Nov 03 22:13:11.687199 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:11.687036    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Nov 03 22:13:11.687199 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:11.687109    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-5qnjg\""
Nov 03 22:13:11.697069 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:11.697048    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4qrls_309c2503-3fef-4048-af01-6b3b70aa52a1/dns/0.log"
Nov 03 22:13:11.713239 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:11.713194    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:13:11.813602 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:11.813571    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/dfa4f1ab-8da9-412a-9e83-b655e65c4212-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-f7ksv\" (UID: \"dfa4f1ab-8da9-412a-9e83-b655e65c4212\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-f7ksv"
Nov 03 22:13:11.813763 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:11.813608    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dfa4f1ab-8da9-412a-9e83-b655e65c4212-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-f7ksv\" (UID: \"dfa4f1ab-8da9-412a-9e83-b655e65c4212\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-f7ksv"
Nov 03 22:13:11.813763 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:11.813631    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dfa4f1ab-8da9-412a-9e83-b655e65c4212-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-f7ksv\" (UID: \"dfa4f1ab-8da9-412a-9e83-b655e65c4212\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-f7ksv"
Nov 03 22:13:11.813841 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:11.813812    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m795\" (UniqueName: \"kubernetes.io/projected/dfa4f1ab-8da9-412a-9e83-b655e65c4212-kube-api-access-2m795\") pod \"prometheus-operator-9cd89b74c-f7ksv\" (UID: \"dfa4f1ab-8da9-412a-9e83-b655e65c4212\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-f7ksv"
Nov 03 22:13:11.853482 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:11.853406    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:13:11.897885 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:11.897856    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4qrls_309c2503-3fef-4048-af01-6b3b70aa52a1/kube-rbac-proxy/0.log"
Nov 03 22:13:11.914959 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:11.914927    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m795\" (UniqueName: \"kubernetes.io/projected/dfa4f1ab-8da9-412a-9e83-b655e65c4212-kube-api-access-2m795\") pod \"prometheus-operator-9cd89b74c-f7ksv\" (UID: \"dfa4f1ab-8da9-412a-9e83-b655e65c4212\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-f7ksv"
Nov 03 22:13:11.915110 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:11.914974    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/dfa4f1ab-8da9-412a-9e83-b655e65c4212-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-f7ksv\" (UID: \"dfa4f1ab-8da9-412a-9e83-b655e65c4212\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-f7ksv"
Nov 03 22:13:11.915372 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:11.915346    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dfa4f1ab-8da9-412a-9e83-b655e65c4212-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-f7ksv\" (UID: \"dfa4f1ab-8da9-412a-9e83-b655e65c4212\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-f7ksv"
Nov 03 22:13:11.915515 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:11.915390    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dfa4f1ab-8da9-412a-9e83-b655e65c4212-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-f7ksv\" (UID: \"dfa4f1ab-8da9-412a-9e83-b655e65c4212\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-f7ksv"
Nov 03 22:13:11.916060 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:11.916036    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dfa4f1ab-8da9-412a-9e83-b655e65c4212-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-f7ksv\" (UID: \"dfa4f1ab-8da9-412a-9e83-b655e65c4212\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-f7ksv"
Nov 03 22:13:11.917997 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:11.917972    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dfa4f1ab-8da9-412a-9e83-b655e65c4212-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-f7ksv\" (UID: \"dfa4f1ab-8da9-412a-9e83-b655e65c4212\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-f7ksv"
Nov 03 22:13:11.921701 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:11.921676    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/dfa4f1ab-8da9-412a-9e83-b655e65c4212-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-f7ksv\" (UID: \"dfa4f1ab-8da9-412a-9e83-b655e65c4212\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-f7ksv"
Nov 03 22:13:11.938973 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:11.938939    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m795\" (UniqueName: \"kubernetes.io/projected/dfa4f1ab-8da9-412a-9e83-b655e65c4212-kube-api-access-2m795\") pod \"prometheus-operator-9cd89b74c-f7ksv\" (UID: \"dfa4f1ab-8da9-412a-9e83-b655e65c4212\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-f7ksv"
Nov 03 22:13:11.968854 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:11.968806    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-f7ksv"
Nov 03 22:13:12.271884 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:12.271841    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-f7ksv"]
Nov 03 22:13:12.276284 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:13:12.276256    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfa4f1ab_8da9_412a_9e83_b655e65c4212.slice/crio-5a7d21e234d308df9d230bd2238b06088f98baf03c5dd7295cc696cbd1c26394 WatchSource:0}: Error finding container 5a7d21e234d308df9d230bd2238b06088f98baf03c5dd7295cc696cbd1c26394: Status 404 returned error can't find the container with id 5a7d21e234d308df9d230bd2238b06088f98baf03c5dd7295cc696cbd1c26394
Nov 03 22:13:12.899705 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:12.899666    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-x7cpm_94868020-2e98-4295-8db5-043461cc6e9d/dns-node-resolver/0.log"
Nov 03 22:13:13.182539 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:13.182502    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-f7ksv" event={"ID":"dfa4f1ab-8da9-412a-9e83-b655e65c4212","Type":"ContainerStarted","Data":"5a7d21e234d308df9d230bd2238b06088f98baf03c5dd7295cc696cbd1c26394"}
Nov 03 22:13:13.496823 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:13.496748    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-fc6bcdddf-9zl84_1bdb4a48-dd3c-4037-8130-78c55a7121b5/registry/0.log"
Nov 03 22:13:13.822843 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:13.822761    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:13:13.961248 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:13.961174    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:13:14.096029 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:14.095953    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-p4p25_cc66a3de-7e7e-4855-8f17-db97dcfa815a/node-ca/0.log"
Nov 03 22:13:14.189102 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:14.189052    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-f7ksv" event={"ID":"dfa4f1ab-8da9-412a-9e83-b655e65c4212","Type":"ContainerStarted","Data":"bee09288874fcee061ac84265573900b7ff732cf3915db640655a87c26d39622"}
Nov 03 22:13:14.189322 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:14.189167    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-f7ksv" event={"ID":"dfa4f1ab-8da9-412a-9e83-b655e65c4212","Type":"ContainerStarted","Data":"c148d21a7561a5026c635591933bd862fa9c729265c2ecb67a1b2aaa78c4fa02"}
Nov 03 22:13:14.231766 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:14.231710    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-f7ksv" podStartSLOduration=1.992399222 podStartE2EDuration="3.231694795s" totalImagesPullingTime="1.239295573s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:13:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:13:14.231310002 +0000 UTC m=+18.257165865" watchObservedRunningTime="2026-11-03 22:13:14.231694795 +0000 UTC m=+18.257550659"
Nov 03 22:13:14.458386 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:14.458346    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:13:14.696636 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:14.696610    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vhccw_365a1938-752c-4cec-a917-c5a46c719307/serve-healthcheck-canary/0.log"
Nov 03 22:13:16.031262 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:16.031211    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:13:16.233139 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:16.233104    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:13:16.422849 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:16.422806    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:13:16.430467 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:16.430433    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-mq2bl"]
Nov 03 22:13:16.433337 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:16.433312    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mq2bl"
Nov 03 22:13:16.459726 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:16.459697    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 03 22:13:16.460612 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:16.460566    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 03 22:13:16.460612 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:16.460597    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 03 22:13:16.460744 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:16.460699    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xxldp\""
Nov 03 22:13:16.551677 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:16.551637    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/97666ecb-018e-46f6-9a80-c0beb294c7d6-root\") pod \"node-exporter-mq2bl\" (UID: \"97666ecb-018e-46f6-9a80-c0beb294c7d6\") " pod="openshift-monitoring/node-exporter-mq2bl"
Nov 03 22:13:16.551838 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:16.551731    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/97666ecb-018e-46f6-9a80-c0beb294c7d6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mq2bl\" (UID: \"97666ecb-018e-46f6-9a80-c0beb294c7d6\") " pod="openshift-monitoring/node-exporter-mq2bl"
Nov 03 22:13:16.551838 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:16.551765    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/97666ecb-018e-46f6-9a80-c0beb294c7d6-node-exporter-wtmp\") pod \"node-exporter-mq2bl\" (UID: \"97666ecb-018e-46f6-9a80-c0beb294c7d6\") " pod="openshift-monitoring/node-exporter-mq2bl"
Nov 03 22:13:16.551838 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:16.551832    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnnhm\" (UniqueName: \"kubernetes.io/projected/97666ecb-018e-46f6-9a80-c0beb294c7d6-kube-api-access-xnnhm\") pod \"node-exporter-mq2bl\" (UID: \"97666ecb-018e-46f6-9a80-c0beb294c7d6\") " pod="openshift-monitoring/node-exporter-mq2bl"
Nov 03 22:13:16.551982 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:16.551856    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/97666ecb-018e-46f6-9a80-c0beb294c7d6-node-exporter-accelerators-collector-config\") pod \"node-exporter-mq2bl\" (UID: \"97666ecb-018e-46f6-9a80-c0beb294c7d6\") " pod="openshift-monitoring/node-exporter-mq2bl"
Nov 03 22:13:16.551982 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:16.551881    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/97666ecb-018e-46f6-9a80-c0beb294c7d6-node-exporter-textfile\") pod \"node-exporter-mq2bl\" (UID: \"97666ecb-018e-46f6-9a80-c0beb294c7d6\") " pod="openshift-monitoring/node-exporter-mq2bl"
Nov 03 22:13:16.551982 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:16.551907    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/97666ecb-018e-46f6-9a80-c0beb294c7d6-sys\") pod \"node-exporter-mq2bl\" (UID: \"97666ecb-018e-46f6-9a80-c0beb294c7d6\") " pod="openshift-monitoring/node-exporter-mq2bl"
Nov 03 22:13:16.551982 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:16.551978    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/97666ecb-018e-46f6-9a80-c0beb294c7d6-metrics-client-ca\") pod \"node-exporter-mq2bl\" (UID: \"97666ecb-018e-46f6-9a80-c0beb294c7d6\") " pod="openshift-monitoring/node-exporter-mq2bl"
Nov 03 22:13:16.552176 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:16.552006    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/97666ecb-018e-46f6-9a80-c0beb294c7d6-node-exporter-tls\") pod \"node-exporter-mq2bl\" (UID: \"97666ecb-018e-46f6-9a80-c0beb294c7d6\") " pod="openshift-monitoring/node-exporter-mq2bl"
Nov 03 22:13:16.564326 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:16.564295    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:13:16.652485 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:16.652453    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/97666ecb-018e-46f6-9a80-c0beb294c7d6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mq2bl\" (UID: \"97666ecb-018e-46f6-9a80-c0beb294c7d6\") " pod="openshift-monitoring/node-exporter-mq2bl"
Nov 03 22:13:16.652651 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:16.652497    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/97666ecb-018e-46f6-9a80-c0beb294c7d6-node-exporter-wtmp\") pod \"node-exporter-mq2bl\" (UID: \"97666ecb-018e-46f6-9a80-c0beb294c7d6\") " pod="openshift-monitoring/node-exporter-mq2bl"
Nov 03 22:13:16.652651 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:16.652638    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/97666ecb-018e-46f6-9a80-c0beb294c7d6-node-exporter-wtmp\") pod \"node-exporter-mq2bl\" (UID: \"97666ecb-018e-46f6-9a80-c0beb294c7d6\") " pod="openshift-monitoring/node-exporter-mq2bl"
Nov 03 22:13:16.652765 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:16.652674    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnnhm\" (UniqueName: \"kubernetes.io/projected/97666ecb-018e-46f6-9a80-c0beb294c7d6-kube-api-access-xnnhm\") pod \"node-exporter-mq2bl\" (UID: \"97666ecb-018e-46f6-9a80-c0beb294c7d6\") " pod="openshift-monitoring/node-exporter-mq2bl"
Nov 03 22:13:16.652765 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:16.652705    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/97666ecb-018e-46f6-9a80-c0beb294c7d6-node-exporter-accelerators-collector-config\") pod \"node-exporter-mq2bl\" (UID: \"97666ecb-018e-46f6-9a80-c0beb294c7d6\") " pod="openshift-monitoring/node-exporter-mq2bl"
Nov 03 22:13:16.652765 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:16.652735    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/97666ecb-018e-46f6-9a80-c0beb294c7d6-node-exporter-textfile\") pod \"node-exporter-mq2bl\" (UID: \"97666ecb-018e-46f6-9a80-c0beb294c7d6\") " pod="openshift-monitoring/node-exporter-mq2bl"
Nov 03 22:13:16.652765 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:16.652756    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/97666ecb-018e-46f6-9a80-c0beb294c7d6-sys\") pod \"node-exporter-mq2bl\" (UID: \"97666ecb-018e-46f6-9a80-c0beb294c7d6\") " pod="openshift-monitoring/node-exporter-mq2bl"
Nov 03 22:13:16.652936 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:16.652777    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/97666ecb-018e-46f6-9a80-c0beb294c7d6-metrics-client-ca\") pod \"node-exporter-mq2bl\" (UID: \"97666ecb-018e-46f6-9a80-c0beb294c7d6\") " pod="openshift-monitoring/node-exporter-mq2bl"
Nov 03 22:13:16.652936 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:16.652801    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/97666ecb-018e-46f6-9a80-c0beb294c7d6-node-exporter-tls\") pod \"node-exporter-mq2bl\" (UID: \"97666ecb-018e-46f6-9a80-c0beb294c7d6\") " pod="openshift-monitoring/node-exporter-mq2bl"
Nov 03 22:13:16.652936 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:16.652838    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/97666ecb-018e-46f6-9a80-c0beb294c7d6-root\") pod \"node-exporter-mq2bl\" (UID: \"97666ecb-018e-46f6-9a80-c0beb294c7d6\") " pod="openshift-monitoring/node-exporter-mq2bl"
Nov 03 22:13:16.653075 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:16.652915    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/97666ecb-018e-46f6-9a80-c0beb294c7d6-root\") pod \"node-exporter-mq2bl\" (UID: \"97666ecb-018e-46f6-9a80-c0beb294c7d6\") " pod="openshift-monitoring/node-exporter-mq2bl"
Nov 03 22:13:16.653075 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:16.652978    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/97666ecb-018e-46f6-9a80-c0beb294c7d6-sys\") pod \"node-exporter-mq2bl\" (UID: \"97666ecb-018e-46f6-9a80-c0beb294c7d6\") " pod="openshift-monitoring/node-exporter-mq2bl"
Nov 03 22:13:16.653252 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:16.653229    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/97666ecb-018e-46f6-9a80-c0beb294c7d6-node-exporter-textfile\") pod \"node-exporter-mq2bl\" (UID: \"97666ecb-018e-46f6-9a80-c0beb294c7d6\") " pod="openshift-monitoring/node-exporter-mq2bl"
Nov 03 22:13:16.653569 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:16.653531    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/97666ecb-018e-46f6-9a80-c0beb294c7d6-metrics-client-ca\") pod \"node-exporter-mq2bl\" (UID: \"97666ecb-018e-46f6-9a80-c0beb294c7d6\") " pod="openshift-monitoring/node-exporter-mq2bl"
Nov 03 22:13:16.653642 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:16.653620    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/97666ecb-018e-46f6-9a80-c0beb294c7d6-node-exporter-accelerators-collector-config\") pod \"node-exporter-mq2bl\" (UID: \"97666ecb-018e-46f6-9a80-c0beb294c7d6\") " pod="openshift-monitoring/node-exporter-mq2bl"
Nov 03 22:13:16.655148 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:16.655123    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/97666ecb-018e-46f6-9a80-c0beb294c7d6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mq2bl\" (UID: \"97666ecb-018e-46f6-9a80-c0beb294c7d6\") " pod="openshift-monitoring/node-exporter-mq2bl"
Nov 03 22:13:16.655512 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:16.655495    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/97666ecb-018e-46f6-9a80-c0beb294c7d6-node-exporter-tls\") pod \"node-exporter-mq2bl\" (UID: \"97666ecb-018e-46f6-9a80-c0beb294c7d6\") " pod="openshift-monitoring/node-exporter-mq2bl"
Nov 03 22:13:16.676799 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:16.676738    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnnhm\" (UniqueName: \"kubernetes.io/projected/97666ecb-018e-46f6-9a80-c0beb294c7d6-kube-api-access-xnnhm\") pod \"node-exporter-mq2bl\" (UID: \"97666ecb-018e-46f6-9a80-c0beb294c7d6\") " pod="openshift-monitoring/node-exporter-mq2bl"
Nov 03 22:13:16.737070 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:16.737017    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:13:16.747884 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:16.747851    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mq2bl"
Nov 03 22:13:16.759162 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:13:16.759128    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97666ecb_018e_46f6_9a80_c0beb294c7d6.slice/crio-c3a112db55ef031f91e87f8fa1c7f3df287338773174b9c3bea7f8dd353b6cde WatchSource:0}: Error finding container c3a112db55ef031f91e87f8fa1c7f3df287338773174b9c3bea7f8dd353b6cde: Status 404 returned error can't find the container with id c3a112db55ef031f91e87f8fa1c7f3df287338773174b9c3bea7f8dd353b6cde
Nov 03 22:13:16.878899 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:16.878860    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:13:17.035564 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:17.035475    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:13:17.158572 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:17.158538    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-4qrls"
Nov 03 22:13:17.172626 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:17.172595    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:13:17.202771 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:17.202726    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mq2bl" event={"ID":"97666ecb-018e-46f6-9a80-c0beb294c7d6","Type":"ContainerStarted","Data":"c3a112db55ef031f91e87f8fa1c7f3df287338773174b9c3bea7f8dd353b6cde"}
Nov 03 22:13:17.315920 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:17.315747    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:13:17.454823 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:17.454771    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:13:17.593598 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:17.593526    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:13:17.655330 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:17.655292    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 22:13:17.658209 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:17.658186    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 22:13:17.679903 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:17.679879    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 03 22:13:17.680069 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:17.680001    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 03 22:13:17.680069 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:17.680055    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 03 22:13:17.680192 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:17.680074    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 03 22:13:17.680192 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:17.680119    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 03 22:13:17.680192 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:17.680149    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-pw7hg\""
Nov 03 22:13:17.680192 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:17.680171    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 03 22:13:17.706418 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:17.706387    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 03 22:13:17.706542 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:17.706387    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 03 22:13:17.707701 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:17.707673    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 22:13:17.735715 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:17.735685    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:13:17.760934 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:17.760897    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/43af3693-5cfe-4142-8880-b448d255c9a2-config-volume\") pod \"alertmanager-main-0\" (UID: \"43af3693-5cfe-4142-8880-b448d255c9a2\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 22:13:17.761102 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:17.760946    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/43af3693-5cfe-4142-8880-b448d255c9a2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"43af3693-5cfe-4142-8880-b448d255c9a2\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 22:13:17.761102 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:17.761038    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/43af3693-5cfe-4142-8880-b448d255c9a2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"43af3693-5cfe-4142-8880-b448d255c9a2\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 22:13:17.761102 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:17.761076    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43af3693-5cfe-4142-8880-b448d255c9a2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"43af3693-5cfe-4142-8880-b448d255c9a2\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 22:13:17.761286 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:17.761133    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/43af3693-5cfe-4142-8880-b448d255c9a2-config-out\") pod \"alertmanager-main-0\" (UID: \"43af3693-5cfe-4142-8880-b448d255c9a2\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 22:13:17.761286 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:17.761167    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43af3693-5cfe-4142-8880-b448d255c9a2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"43af3693-5cfe-4142-8880-b448d255c9a2\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 22:13:17.761286 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:17.761193    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/43af3693-5cfe-4142-8880-b448d255c9a2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"43af3693-5cfe-4142-8880-b448d255c9a2\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 22:13:17.761286 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:17.761254    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/43af3693-5cfe-4142-8880-b448d255c9a2-web-config\") pod \"alertmanager-main-0\" (UID: \"43af3693-5cfe-4142-8880-b448d255c9a2\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 22:13:17.761494 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:17.761283    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/43af3693-5cfe-4142-8880-b448d255c9a2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"43af3693-5cfe-4142-8880-b448d255c9a2\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 22:13:17.761494 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:17.761315    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q95nl\" (UniqueName: \"kubernetes.io/projected/43af3693-5cfe-4142-8880-b448d255c9a2-kube-api-access-q95nl\") pod \"alertmanager-main-0\" (UID: \"43af3693-5cfe-4142-8880-b448d255c9a2\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 22:13:17.761494 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:17.761364    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/43af3693-5cfe-4142-8880-b448d255c9a2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"43af3693-5cfe-4142-8880-b448d255c9a2\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 22:13:17.761494 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:17.761388    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/43af3693-5cfe-4142-8880-b448d255c9a2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"43af3693-5cfe-4142-8880-b448d255c9a2\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 22:13:17.761494 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:17.761417    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/43af3693-5cfe-4142-8880-b448d255c9a2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"43af3693-5cfe-4142-8880-b448d255c9a2\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 22:13:17.784451 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:17.784419    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 03 22:13:17.863910 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:17.862751    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/43af3693-5cfe-4142-8880-b448d255c9a2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"43af3693-5cfe-4142-8880-b448d255c9a2\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 22:13:17.863910 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:17.862801    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q95nl\" (UniqueName: \"kubernetes.io/projected/43af3693-5cfe-4142-8880-b448d255c9a2-kube-api-access-q95nl\") pod \"alertmanager-main-0\" (UID: \"43af3693-5cfe-4142-8880-b448d255c9a2\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 22:13:17.863910 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:17.862843    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/43af3693-5cfe-4142-8880-b448d255c9a2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"43af3693-5cfe-4142-8880-b448d255c9a2\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 22:13:17.863910 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:17.862864    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/43af3693-5cfe-4142-8880-b448d255c9a2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"43af3693-5cfe-4142-8880-b448d255c9a2\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 22:13:17.863910 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:17.862890    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/43af3693-5cfe-4142-8880-b448d255c9a2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"43af3693-5cfe-4142-8880-b448d255c9a2\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 22:13:17.863910 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:17.862919    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/43af3693-5cfe-4142-8880-b448d255c9a2-config-volume\") pod \"alertmanager-main-0\" (UID: \"43af3693-5cfe-4142-8880-b448d255c9a2\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 22:13:17.863910 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:17.862941    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/43af3693-5cfe-4142-8880-b448d255c9a2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"43af3693-5cfe-4142-8880-b448d255c9a2\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 22:13:17.863910 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:17.862970    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/43af3693-5cfe-4142-8880-b448d255c9a2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"43af3693-5cfe-4142-8880-b448d255c9a2\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 22:13:17.863910 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:17.862993    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43af3693-5cfe-4142-8880-b448d255c9a2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"43af3693-5cfe-4142-8880-b448d255c9a2\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 22:13:17.863910 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:17.863042    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/43af3693-5cfe-4142-8880-b448d255c9a2-config-out\") pod \"alertmanager-main-0\" (UID: \"43af3693-5cfe-4142-8880-b448d255c9a2\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 22:13:17.863910 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:17.863071    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43af3693-5cfe-4142-8880-b448d255c9a2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"43af3693-5cfe-4142-8880-b448d255c9a2\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 22:13:17.863910 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:17.863109    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/43af3693-5cfe-4142-8880-b448d255c9a2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"43af3693-5cfe-4142-8880-b448d255c9a2\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 22:13:17.863910 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:17.863134    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/43af3693-5cfe-4142-8880-b448d255c9a2-web-config\") pod \"alertmanager-main-0\" (UID: \"43af3693-5cfe-4142-8880-b448d255c9a2\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 22:13:17.865904 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:17.865264    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/43af3693-5cfe-4142-8880-b448d255c9a2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"43af3693-5cfe-4142-8880-b448d255c9a2\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 22:13:17.865904 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:17.865893    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43af3693-5cfe-4142-8880-b448d255c9a2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"43af3693-5cfe-4142-8880-b448d255c9a2\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 22:13:17.866097 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:17.866004    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43af3693-5cfe-4142-8880-b448d255c9a2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"43af3693-5cfe-4142-8880-b448d255c9a2\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 22:13:17.867117 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:17.866607    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/43af3693-5cfe-4142-8880-b448d255c9a2-config-volume\") pod \"alertmanager-main-0\" (UID: \"43af3693-5cfe-4142-8880-b448d255c9a2\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 22:13:17.867117 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:17.866833    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/43af3693-5cfe-4142-8880-b448d255c9a2-web-config\") pod \"alertmanager-main-0\" (UID: \"43af3693-5cfe-4142-8880-b448d255c9a2\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 22:13:17.867117 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:17.867067    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/43af3693-5cfe-4142-8880-b448d255c9a2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"43af3693-5cfe-4142-8880-b448d255c9a2\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 22:13:17.867343 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:17.867117    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/43af3693-5cfe-4142-8880-b448d255c9a2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"43af3693-5cfe-4142-8880-b448d255c9a2\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 22:13:17.867574 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:17.867548    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/43af3693-5cfe-4142-8880-b448d255c9a2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"43af3693-5cfe-4142-8880-b448d255c9a2\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 22:13:17.867922 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:17.867895    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/43af3693-5cfe-4142-8880-b448d255c9a2-config-out\") pod \"alertmanager-main-0\" (UID: \"43af3693-5cfe-4142-8880-b448d255c9a2\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 22:13:17.868368 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:17.868346    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/43af3693-5cfe-4142-8880-b448d255c9a2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"43af3693-5cfe-4142-8880-b448d255c9a2\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 22:13:17.868429 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:17.868393    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/43af3693-5cfe-4142-8880-b448d255c9a2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"43af3693-5cfe-4142-8880-b448d255c9a2\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 22:13:17.868798 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:17.868775    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/43af3693-5cfe-4142-8880-b448d255c9a2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"43af3693-5cfe-4142-8880-b448d255c9a2\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 22:13:17.889063 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:17.889037    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q95nl\" (UniqueName: \"kubernetes.io/projected/43af3693-5cfe-4142-8880-b448d255c9a2-kube-api-access-q95nl\") pod \"alertmanager-main-0\" (UID: \"43af3693-5cfe-4142-8880-b448d255c9a2\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 22:13:17.979000 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:17.978963    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 22:13:18.208224 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:18.208188    4431 generic.go:356] "Generic (PLEG): container finished" podID="97666ecb-018e-46f6-9a80-c0beb294c7d6" containerID="ca4f16fd78b7c93de5e653a99dcd9b0ec36aa2a71a7186f76c00b6c56c9c9a87" exitCode=0
Nov 03 22:13:18.208646 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:18.208291    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mq2bl" event={"ID":"97666ecb-018e-46f6-9a80-c0beb294c7d6","Type":"ContainerDied","Data":"ca4f16fd78b7c93de5e653a99dcd9b0ec36aa2a71a7186f76c00b6c56c9c9a87"}
Nov 03 22:13:18.313982 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:18.313889    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 22:13:19.214623 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:19.214533    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mq2bl" event={"ID":"97666ecb-018e-46f6-9a80-c0beb294c7d6","Type":"ContainerStarted","Data":"4f2e718552925c1adc74bca2b578c40c75623ea1f44665012c0d7dfa7100f818"}
Nov 03 22:13:19.214899 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:19.214639    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mq2bl" event={"ID":"97666ecb-018e-46f6-9a80-c0beb294c7d6","Type":"ContainerStarted","Data":"8b67496e61e57b0d56f5a155fce317abd7c96a22265c1abf24b8954b680cb2c0"}
Nov 03 22:13:19.216934 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:19.216903    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"43af3693-5cfe-4142-8880-b448d255c9a2","Type":"ContainerStarted","Data":"764eab8e9eee089c1ffbd8373e72f091286b43f93ce54f4b167ea338280b278b"}
Nov 03 22:13:19.258963 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:19.258908    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-mq2bl" podStartSLOduration=2.276294977 podStartE2EDuration="3.258890954s" totalImagesPullingTime="901.776951ms" totalInitContainerRuntime="80.819026ms" isStatefulPod=true podCreationTimestamp="2026-11-03 22:13:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:13:19.258076269 +0000 UTC m=+23.283932145" watchObservedRunningTime="2026-11-03 22:13:19.258890954 +0000 UTC m=+23.284746811"
Nov 03 22:13:19.593305 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:19.593207    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:13:19.742158 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:19.742115    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:13:19.916208 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:19.916174    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:13:20.056901 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:20.056854    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:13:20.221834 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:20.221759    4431 generic.go:356] "Generic (PLEG): container finished" podID="43af3693-5cfe-4142-8880-b448d255c9a2" containerID="4c45119c994eca23755699862d79d9b93fa7850e8018ec39c656087efe1fc13d" exitCode=0
Nov 03 22:13:20.222258 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:20.221834    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"43af3693-5cfe-4142-8880-b448d255c9a2","Type":"ContainerDied","Data":"4c45119c994eca23755699862d79d9b93fa7850e8018ec39c656087efe1fc13d"}
Nov 03 22:13:20.709891 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:20.709852    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:13:20.853954 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:20.853915    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:13:21.117969 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:21.117883    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:13:21.257200 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:21.257152    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:13:21.397016 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:21.396936    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:13:21.407491 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:21.407461    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-7p8pl"
Nov 03 22:13:21.666108 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:21.666072    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-979f6bd4f-xgz4d"]
Nov 03 22:13:21.668752 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:21.668726    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-979f6bd4f-xgz4d"
Nov 03 22:13:21.684698 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:21.684671    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-979f6bd4f-xgz4d"]
Nov 03 22:13:21.696858 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:21.696138    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-lmv7x\""
Nov 03 22:13:21.696858 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:21.696246    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Nov 03 22:13:21.696858 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:21.696447    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Nov 03 22:13:21.708693 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:21.708672    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Nov 03 22:13:21.709903 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:21.709699    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Nov 03 22:13:21.709903 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:21.709837    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Nov 03 22:13:21.710274 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:21.710252    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Nov 03 22:13:21.800425 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:21.800388    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/66a72772-83f1-4117-a856-f52c0ebbe966-telemeter-client-tls\") pod \"telemeter-client-979f6bd4f-xgz4d\" (UID: \"66a72772-83f1-4117-a856-f52c0ebbe966\") " pod="openshift-monitoring/telemeter-client-979f6bd4f-xgz4d"
Nov 03 22:13:21.800599 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:21.800434    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/66a72772-83f1-4117-a856-f52c0ebbe966-secret-telemeter-client\") pod \"telemeter-client-979f6bd4f-xgz4d\" (UID: \"66a72772-83f1-4117-a856-f52c0ebbe966\") " pod="openshift-monitoring/telemeter-client-979f6bd4f-xgz4d"
Nov 03 22:13:21.800599 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:21.800469    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66a72772-83f1-4117-a856-f52c0ebbe966-serving-certs-ca-bundle\") pod \"telemeter-client-979f6bd4f-xgz4d\" (UID: \"66a72772-83f1-4117-a856-f52c0ebbe966\") " pod="openshift-monitoring/telemeter-client-979f6bd4f-xgz4d"
Nov 03 22:13:21.800599 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:21.800545    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/66a72772-83f1-4117-a856-f52c0ebbe966-federate-client-tls\") pod \"telemeter-client-979f6bd4f-xgz4d\" (UID: \"66a72772-83f1-4117-a856-f52c0ebbe966\") " pod="openshift-monitoring/telemeter-client-979f6bd4f-xgz4d"
Nov 03 22:13:21.800758 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:21.800616    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dmgx\" (UniqueName: \"kubernetes.io/projected/66a72772-83f1-4117-a856-f52c0ebbe966-kube-api-access-7dmgx\") pod \"telemeter-client-979f6bd4f-xgz4d\" (UID: \"66a72772-83f1-4117-a856-f52c0ebbe966\") " pod="openshift-monitoring/telemeter-client-979f6bd4f-xgz4d"
Nov 03 22:13:21.800758 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:21.800640    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66a72772-83f1-4117-a856-f52c0ebbe966-telemeter-trusted-ca-bundle\") pod \"telemeter-client-979f6bd4f-xgz4d\" (UID: \"66a72772-83f1-4117-a856-f52c0ebbe966\") " pod="openshift-monitoring/telemeter-client-979f6bd4f-xgz4d"
Nov 03 22:13:21.800758 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:21.800731    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/66a72772-83f1-4117-a856-f52c0ebbe966-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-979f6bd4f-xgz4d\" (UID: \"66a72772-83f1-4117-a856-f52c0ebbe966\") " pod="openshift-monitoring/telemeter-client-979f6bd4f-xgz4d"
Nov 03 22:13:21.800909 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:21.800771    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/66a72772-83f1-4117-a856-f52c0ebbe966-metrics-client-ca\") pod \"telemeter-client-979f6bd4f-xgz4d\" (UID: \"66a72772-83f1-4117-a856-f52c0ebbe966\") " pod="openshift-monitoring/telemeter-client-979f6bd4f-xgz4d"
Nov 03 22:13:21.901553 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:21.901526    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/66a72772-83f1-4117-a856-f52c0ebbe966-metrics-client-ca\") pod \"telemeter-client-979f6bd4f-xgz4d\" (UID: \"66a72772-83f1-4117-a856-f52c0ebbe966\") " pod="openshift-monitoring/telemeter-client-979f6bd4f-xgz4d"
Nov 03 22:13:21.901665 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:21.901574    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/66a72772-83f1-4117-a856-f52c0ebbe966-telemeter-client-tls\") pod \"telemeter-client-979f6bd4f-xgz4d\" (UID: \"66a72772-83f1-4117-a856-f52c0ebbe966\") " pod="openshift-monitoring/telemeter-client-979f6bd4f-xgz4d"
Nov 03 22:13:21.901665 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:21.901612    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/66a72772-83f1-4117-a856-f52c0ebbe966-secret-telemeter-client\") pod \"telemeter-client-979f6bd4f-xgz4d\" (UID: \"66a72772-83f1-4117-a856-f52c0ebbe966\") " pod="openshift-monitoring/telemeter-client-979f6bd4f-xgz4d"
Nov 03 22:13:21.901665 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:21.901650    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66a72772-83f1-4117-a856-f52c0ebbe966-serving-certs-ca-bundle\") pod \"telemeter-client-979f6bd4f-xgz4d\" (UID: \"66a72772-83f1-4117-a856-f52c0ebbe966\") " pod="openshift-monitoring/telemeter-client-979f6bd4f-xgz4d"
Nov 03 22:13:21.901765 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:21.901675    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/66a72772-83f1-4117-a856-f52c0ebbe966-federate-client-tls\") pod \"telemeter-client-979f6bd4f-xgz4d\" (UID: \"66a72772-83f1-4117-a856-f52c0ebbe966\") " pod="openshift-monitoring/telemeter-client-979f6bd4f-xgz4d"
Nov 03 22:13:21.901765 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:21.901704    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dmgx\" (UniqueName: \"kubernetes.io/projected/66a72772-83f1-4117-a856-f52c0ebbe966-kube-api-access-7dmgx\") pod \"telemeter-client-979f6bd4f-xgz4d\" (UID: \"66a72772-83f1-4117-a856-f52c0ebbe966\") " pod="openshift-monitoring/telemeter-client-979f6bd4f-xgz4d"
Nov 03 22:13:21.901765 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:21.901735    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66a72772-83f1-4117-a856-f52c0ebbe966-telemeter-trusted-ca-bundle\") pod \"telemeter-client-979f6bd4f-xgz4d\" (UID: \"66a72772-83f1-4117-a856-f52c0ebbe966\") " pod="openshift-monitoring/telemeter-client-979f6bd4f-xgz4d"
Nov 03 22:13:21.901849 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:21.901787    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/66a72772-83f1-4117-a856-f52c0ebbe966-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-979f6bd4f-xgz4d\" (UID: \"66a72772-83f1-4117-a856-f52c0ebbe966\") " pod="openshift-monitoring/telemeter-client-979f6bd4f-xgz4d"
Nov 03 22:13:21.902295 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:21.902272    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/66a72772-83f1-4117-a856-f52c0ebbe966-metrics-client-ca\") pod \"telemeter-client-979f6bd4f-xgz4d\" (UID: \"66a72772-83f1-4117-a856-f52c0ebbe966\") " pod="openshift-monitoring/telemeter-client-979f6bd4f-xgz4d"
Nov 03 22:13:21.902791 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:21.902762    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66a72772-83f1-4117-a856-f52c0ebbe966-serving-certs-ca-bundle\") pod \"telemeter-client-979f6bd4f-xgz4d\" (UID: \"66a72772-83f1-4117-a856-f52c0ebbe966\") " pod="openshift-monitoring/telemeter-client-979f6bd4f-xgz4d"
Nov 03 22:13:21.903111 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:21.903090    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66a72772-83f1-4117-a856-f52c0ebbe966-telemeter-trusted-ca-bundle\") pod \"telemeter-client-979f6bd4f-xgz4d\" (UID: \"66a72772-83f1-4117-a856-f52c0ebbe966\") " pod="openshift-monitoring/telemeter-client-979f6bd4f-xgz4d"
Nov 03 22:13:21.906094 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:21.906069    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/66a72772-83f1-4117-a856-f52c0ebbe966-telemeter-client-tls\") pod \"telemeter-client-979f6bd4f-xgz4d\" (UID: \"66a72772-83f1-4117-a856-f52c0ebbe966\") " pod="openshift-monitoring/telemeter-client-979f6bd4f-xgz4d"
Nov 03 22:13:21.907262 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:21.907210    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/66a72772-83f1-4117-a856-f52c0ebbe966-secret-telemeter-client\") pod \"telemeter-client-979f6bd4f-xgz4d\" (UID: \"66a72772-83f1-4117-a856-f52c0ebbe966\") " pod="openshift-monitoring/telemeter-client-979f6bd4f-xgz4d"
Nov 03 22:13:21.907711 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:21.907686    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/66a72772-83f1-4117-a856-f52c0ebbe966-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-979f6bd4f-xgz4d\" (UID: \"66a72772-83f1-4117-a856-f52c0ebbe966\") " pod="openshift-monitoring/telemeter-client-979f6bd4f-xgz4d"
Nov 03 22:13:21.909287 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:21.909254    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/66a72772-83f1-4117-a856-f52c0ebbe966-federate-client-tls\") pod \"telemeter-client-979f6bd4f-xgz4d\" (UID: \"66a72772-83f1-4117-a856-f52c0ebbe966\") " pod="openshift-monitoring/telemeter-client-979f6bd4f-xgz4d"
Nov 03 22:13:21.926112 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:21.926083    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dmgx\" (UniqueName: \"kubernetes.io/projected/66a72772-83f1-4117-a856-f52c0ebbe966-kube-api-access-7dmgx\") pod \"telemeter-client-979f6bd4f-xgz4d\" (UID: \"66a72772-83f1-4117-a856-f52c0ebbe966\") " pod="openshift-monitoring/telemeter-client-979f6bd4f-xgz4d"
Nov 03 22:13:21.985267 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:21.985237    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-979f6bd4f-xgz4d"
Nov 03 22:13:22.232853 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:22.232777    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"43af3693-5cfe-4142-8880-b448d255c9a2","Type":"ContainerStarted","Data":"6e385664c9850bbc41e00b4bd353aeb782044a695cb066231e3222404dc8b745"}
Nov 03 22:13:22.233006 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:22.232867    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"43af3693-5cfe-4142-8880-b448d255c9a2","Type":"ContainerStarted","Data":"da0d3be69e6ce965f95eb6dbb775cac4b1803b5f27909a36f1e67f6c9075de30"}
Nov 03 22:13:22.233006 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:22.232909    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"43af3693-5cfe-4142-8880-b448d255c9a2","Type":"ContainerStarted","Data":"a50202a2e2777b686132a5906736dce825d122c07868027079b183fac2359132"}
Nov 03 22:13:22.233006 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:22.232951    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"43af3693-5cfe-4142-8880-b448d255c9a2","Type":"ContainerStarted","Data":"6ae87ea73103189bd05e5802ae9a8ded55478d4333bf735c9eac3e0e87799436"}
Nov 03 22:13:22.233006 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:22.232988    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"43af3693-5cfe-4142-8880-b448d255c9a2","Type":"ContainerStarted","Data":"3def159a0f1d846eb61c41295effcae60e18248b8a066e11d4edb17c38f0a6b6"}
Nov 03 22:13:22.326156 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:22.326126    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-979f6bd4f-xgz4d"]
Nov 03 22:13:22.329157 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:13:22.329124    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66a72772_83f1_4117_a856_f52c0ebbe966.slice/crio-e4225896a67051b6a8b2a68b9475473a7982113ebf46d1a77a11540ba3bd1f2d WatchSource:0}: Error finding container e4225896a67051b6a8b2a68b9475473a7982113ebf46d1a77a11540ba3bd1f2d: Status 404 returned error can't find the container with id e4225896a67051b6a8b2a68b9475473a7982113ebf46d1a77a11540ba3bd1f2d
Nov 03 22:13:23.239427 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.239338    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-979f6bd4f-xgz4d" event={"ID":"66a72772-83f1-4117-a856-f52c0ebbe966","Type":"ContainerStarted","Data":"e4225896a67051b6a8b2a68b9475473a7982113ebf46d1a77a11540ba3bd1f2d"}
Nov 03 22:13:23.244101 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.244022    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"43af3693-5cfe-4142-8880-b448d255c9a2","Type":"ContainerStarted","Data":"21795324f5c89751e2ae8369fcd96bf0a88cc561a2a75b5163fac609ca6a06d2"}
Nov 03 22:13:23.336271 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.332892    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.133013473 podStartE2EDuration="6.332872265s" totalImagesPullingTime="3.184103154s" totalInitContainerRuntime="15.755638ms" isStatefulPod=false podCreationTimestamp="2026-11-03 22:13:17 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:13:23.319514863 +0000 UTC m=+27.345370726" watchObservedRunningTime="2026-11-03 22:13:23.332872265 +0000 UTC m=+27.358728129"
Nov 03 22:13:23.336271 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.333878    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 22:13:23.338103 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.338078    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:13:23.355106 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.355075    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Nov 03 22:13:23.355106 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.355120    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Nov 03 22:13:23.355330 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.355249    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 03 22:13:23.355444 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.355426    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Nov 03 22:13:23.355533 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.355518    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Nov 03 22:13:23.355649 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.355632    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Nov 03 22:13:23.355751 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.355739    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Nov 03 22:13:23.355815 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.355803    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Nov 03 22:13:23.355878 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.355867    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Nov 03 22:13:23.355965 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.355947    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Nov 03 22:13:23.372262 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.366669    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Nov 03 22:13:23.372262 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.366670    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-eviee284905oq\""
Nov 03 22:13:23.372262 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.366748    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Nov 03 22:13:23.372262 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.366774    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-2hcp9\""
Nov 03 22:13:23.372262 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.366790    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Nov 03 22:13:23.372262 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.369477    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 22:13:23.372262 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.369784    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Nov 03 22:13:23.372262 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.369945    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Nov 03 22:13:23.372262 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.370747    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:13:23.514804 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.514731    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1e33b576-1208-4325-87e9-4351920bfa06-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1e33b576-1208-4325-87e9-4351920bfa06\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:13:23.514804 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.514766    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1e33b576-1208-4325-87e9-4351920bfa06-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1e33b576-1208-4325-87e9-4351920bfa06\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:13:23.514804 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.514799    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1e33b576-1208-4325-87e9-4351920bfa06-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1e33b576-1208-4325-87e9-4351920bfa06\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:13:23.515042 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.514829    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxpzp\" (UniqueName: \"kubernetes.io/projected/1e33b576-1208-4325-87e9-4351920bfa06-kube-api-access-sxpzp\") pod \"prometheus-k8s-0\" (UID: \"1e33b576-1208-4325-87e9-4351920bfa06\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:13:23.515042 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.514858    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e33b576-1208-4325-87e9-4351920bfa06-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1e33b576-1208-4325-87e9-4351920bfa06\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:13:23.515042 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.514883    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1e33b576-1208-4325-87e9-4351920bfa06-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1e33b576-1208-4325-87e9-4351920bfa06\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:13:23.515042 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.514909    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1e33b576-1208-4325-87e9-4351920bfa06-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1e33b576-1208-4325-87e9-4351920bfa06\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:13:23.515042 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.514935    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1e33b576-1208-4325-87e9-4351920bfa06-config-out\") pod \"prometheus-k8s-0\" (UID: \"1e33b576-1208-4325-87e9-4351920bfa06\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:13:23.515042 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.514950    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1e33b576-1208-4325-87e9-4351920bfa06-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1e33b576-1208-4325-87e9-4351920bfa06\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:13:23.515042 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.514965    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e33b576-1208-4325-87e9-4351920bfa06-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1e33b576-1208-4325-87e9-4351920bfa06\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:13:23.515042 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.514985    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1e33b576-1208-4325-87e9-4351920bfa06-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1e33b576-1208-4325-87e9-4351920bfa06\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:13:23.515042 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.515015    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1e33b576-1208-4325-87e9-4351920bfa06-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1e33b576-1208-4325-87e9-4351920bfa06\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:13:23.515042 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.515034    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1e33b576-1208-4325-87e9-4351920bfa06-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1e33b576-1208-4325-87e9-4351920bfa06\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:13:23.515450 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.515049    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e33b576-1208-4325-87e9-4351920bfa06-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1e33b576-1208-4325-87e9-4351920bfa06\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:13:23.515450 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.515066    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1e33b576-1208-4325-87e9-4351920bfa06-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1e33b576-1208-4325-87e9-4351920bfa06\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:13:23.515450 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.515087    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1e33b576-1208-4325-87e9-4351920bfa06-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1e33b576-1208-4325-87e9-4351920bfa06\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:13:23.515450 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.515111    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1e33b576-1208-4325-87e9-4351920bfa06-web-config\") pod \"prometheus-k8s-0\" (UID: \"1e33b576-1208-4325-87e9-4351920bfa06\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:13:23.515450 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.515128    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1e33b576-1208-4325-87e9-4351920bfa06-config\") pod \"prometheus-k8s-0\" (UID: \"1e33b576-1208-4325-87e9-4351920bfa06\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:13:23.515450 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.515142    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1e33b576-1208-4325-87e9-4351920bfa06-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1e33b576-1208-4325-87e9-4351920bfa06\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:13:23.515450 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.515159    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1e33b576-1208-4325-87e9-4351920bfa06-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1e33b576-1208-4325-87e9-4351920bfa06\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:13:23.621407 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.621325    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1e33b576-1208-4325-87e9-4351920bfa06-config-out\") pod \"prometheus-k8s-0\" (UID: \"1e33b576-1208-4325-87e9-4351920bfa06\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:13:23.621594 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.621417    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1e33b576-1208-4325-87e9-4351920bfa06-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1e33b576-1208-4325-87e9-4351920bfa06\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:13:23.621594 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.621451    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e33b576-1208-4325-87e9-4351920bfa06-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1e33b576-1208-4325-87e9-4351920bfa06\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:13:23.621594 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.621494    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1e33b576-1208-4325-87e9-4351920bfa06-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1e33b576-1208-4325-87e9-4351920bfa06\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:13:23.621594 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.621541    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1e33b576-1208-4325-87e9-4351920bfa06-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1e33b576-1208-4325-87e9-4351920bfa06\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:13:23.621594 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.621576    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1e33b576-1208-4325-87e9-4351920bfa06-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1e33b576-1208-4325-87e9-4351920bfa06\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:13:23.621844 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.621602    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e33b576-1208-4325-87e9-4351920bfa06-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1e33b576-1208-4325-87e9-4351920bfa06\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:13:23.621844 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.621664    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1e33b576-1208-4325-87e9-4351920bfa06-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1e33b576-1208-4325-87e9-4351920bfa06\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:13:23.621844 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.621704    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1e33b576-1208-4325-87e9-4351920bfa06-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1e33b576-1208-4325-87e9-4351920bfa06\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:13:23.621844 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.621744    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1e33b576-1208-4325-87e9-4351920bfa06-web-config\") pod \"prometheus-k8s-0\" (UID: \"1e33b576-1208-4325-87e9-4351920bfa06\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:13:23.621844 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.621768    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1e33b576-1208-4325-87e9-4351920bfa06-config\") pod \"prometheus-k8s-0\" (UID: \"1e33b576-1208-4325-87e9-4351920bfa06\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:13:23.621844 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.621796    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1e33b576-1208-4325-87e9-4351920bfa06-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1e33b576-1208-4325-87e9-4351920bfa06\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:13:23.621844 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.621826    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1e33b576-1208-4325-87e9-4351920bfa06-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1e33b576-1208-4325-87e9-4351920bfa06\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:13:23.622181 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.621889    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1e33b576-1208-4325-87e9-4351920bfa06-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1e33b576-1208-4325-87e9-4351920bfa06\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:13:23.622181 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.621916    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1e33b576-1208-4325-87e9-4351920bfa06-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1e33b576-1208-4325-87e9-4351920bfa06\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:13:23.622181 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.621945    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1e33b576-1208-4325-87e9-4351920bfa06-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1e33b576-1208-4325-87e9-4351920bfa06\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:13:23.622181 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.621973    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxpzp\" (UniqueName: \"kubernetes.io/projected/1e33b576-1208-4325-87e9-4351920bfa06-kube-api-access-sxpzp\") pod \"prometheus-k8s-0\" (UID: \"1e33b576-1208-4325-87e9-4351920bfa06\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:13:23.622181 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.622003    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e33b576-1208-4325-87e9-4351920bfa06-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1e33b576-1208-4325-87e9-4351920bfa06\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:13:23.622181 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.622034    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1e33b576-1208-4325-87e9-4351920bfa06-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1e33b576-1208-4325-87e9-4351920bfa06\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:13:23.623666 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.623029    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1e33b576-1208-4325-87e9-4351920bfa06-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1e33b576-1208-4325-87e9-4351920bfa06\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:13:23.624472 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.624441    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1e33b576-1208-4325-87e9-4351920bfa06-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1e33b576-1208-4325-87e9-4351920bfa06\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:13:23.624613 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.624583    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e33b576-1208-4325-87e9-4351920bfa06-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1e33b576-1208-4325-87e9-4351920bfa06\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:13:23.624765 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.624742    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1e33b576-1208-4325-87e9-4351920bfa06-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1e33b576-1208-4325-87e9-4351920bfa06\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:13:23.626854 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.626715    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:13:23.626854 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.626750    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1e33b576-1208-4325-87e9-4351920bfa06-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1e33b576-1208-4325-87e9-4351920bfa06\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:13:23.626854 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.626806    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1e33b576-1208-4325-87e9-4351920bfa06-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1e33b576-1208-4325-87e9-4351920bfa06\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:13:23.627022 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.626930    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1e33b576-1208-4325-87e9-4351920bfa06-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1e33b576-1208-4325-87e9-4351920bfa06\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:13:23.627155 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.627129    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e33b576-1208-4325-87e9-4351920bfa06-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1e33b576-1208-4325-87e9-4351920bfa06\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:13:23.627410 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.627384    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1e33b576-1208-4325-87e9-4351920bfa06-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1e33b576-1208-4325-87e9-4351920bfa06\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:13:23.627553 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.627530    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1e33b576-1208-4325-87e9-4351920bfa06-web-config\") pod \"prometheus-k8s-0\" (UID: \"1e33b576-1208-4325-87e9-4351920bfa06\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:13:23.627852 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.627830    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1e33b576-1208-4325-87e9-4351920bfa06-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1e33b576-1208-4325-87e9-4351920bfa06\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:13:23.628192 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.628169    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1e33b576-1208-4325-87e9-4351920bfa06-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1e33b576-1208-4325-87e9-4351920bfa06\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:13:23.628614 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.628591    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e33b576-1208-4325-87e9-4351920bfa06-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1e33b576-1208-4325-87e9-4351920bfa06\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:13:23.628686 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.628624    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1e33b576-1208-4325-87e9-4351920bfa06-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1e33b576-1208-4325-87e9-4351920bfa06\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:13:23.631386 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.631358    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1e33b576-1208-4325-87e9-4351920bfa06-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1e33b576-1208-4325-87e9-4351920bfa06\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:13:23.632293 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.632273    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1e33b576-1208-4325-87e9-4351920bfa06-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1e33b576-1208-4325-87e9-4351920bfa06\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:13:23.633622 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.633599    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1e33b576-1208-4325-87e9-4351920bfa06-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1e33b576-1208-4325-87e9-4351920bfa06\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:13:23.635722 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.635697    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1e33b576-1208-4325-87e9-4351920bfa06-config-out\") pod \"prometheus-k8s-0\" (UID: \"1e33b576-1208-4325-87e9-4351920bfa06\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:13:23.635943 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.635924    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1e33b576-1208-4325-87e9-4351920bfa06-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1e33b576-1208-4325-87e9-4351920bfa06\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:13:23.637600 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.637578    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/1e33b576-1208-4325-87e9-4351920bfa06-config\") pod \"prometheus-k8s-0\" (UID: \"1e33b576-1208-4325-87e9-4351920bfa06\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:13:23.645926 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.645900    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxpzp\" (UniqueName: \"kubernetes.io/projected/1e33b576-1208-4325-87e9-4351920bfa06-kube-api-access-sxpzp\") pod \"prometheus-k8s-0\" (UID: \"1e33b576-1208-4325-87e9-4351920bfa06\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:13:23.679436 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.679398    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:13:23.762694 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:23.762665    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:13:24.031365 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:24.031320    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 22:13:24.045405 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:13:24.045369    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e33b576_1208_4325_87e9_4351920bfa06.slice/crio-3ee14ca7ee632401e828940c6d8369458df64a5342e922765e2814e1d62f253d WatchSource:0}: Error finding container 3ee14ca7ee632401e828940c6d8369458df64a5342e922765e2814e1d62f253d: Status 404 returned error can't find the container with id 3ee14ca7ee632401e828940c6d8369458df64a5342e922765e2814e1d62f253d
Nov 03 22:13:24.254338 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:24.254293    4431 generic.go:356] "Generic (PLEG): container finished" podID="1e33b576-1208-4325-87e9-4351920bfa06" containerID="6a43f85ad035f95fed9c6dde29c7d6d0442d39bb05a5929de3d411711c512196" exitCode=0
Nov 03 22:13:24.254618 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:24.254582    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1e33b576-1208-4325-87e9-4351920bfa06","Type":"ContainerDied","Data":"6a43f85ad035f95fed9c6dde29c7d6d0442d39bb05a5929de3d411711c512196"}
Nov 03 22:13:24.254732 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:24.254662    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1e33b576-1208-4325-87e9-4351920bfa06","Type":"ContainerStarted","Data":"3ee14ca7ee632401e828940c6d8369458df64a5342e922765e2814e1d62f253d"}
Nov 03 22:13:25.262764 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:25.262122    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-979f6bd4f-xgz4d" event={"ID":"66a72772-83f1-4117-a856-f52c0ebbe966","Type":"ContainerStarted","Data":"7afda362b29404f889b7bfbaa86146c7cc0bd5305a6b91bd710a57d2716e2370"}
Nov 03 22:13:25.262764 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:25.262191    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-979f6bd4f-xgz4d" event={"ID":"66a72772-83f1-4117-a856-f52c0ebbe966","Type":"ContainerStarted","Data":"4086f5c8a282f4742fb3c5b41761cd0fcf91712664fb88467506d112f86fab85"}
Nov 03 22:13:25.262764 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:25.262241    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-979f6bd4f-xgz4d" event={"ID":"66a72772-83f1-4117-a856-f52c0ebbe966","Type":"ContainerStarted","Data":"662443a787307683b13b3c2c1d9d69a09edcccf0b0550727ab2a596825a6bb78"}
Nov 03 22:13:25.323718 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:25.323659    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-979f6bd4f-xgz4d" podStartSLOduration=2.257131251 podStartE2EDuration="4.323646001s" totalImagesPullingTime="2.06651475s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:13:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:13:25.322658303 +0000 UTC m=+29.348514164" watchObservedRunningTime="2026-11-03 22:13:25.323646001 +0000 UTC m=+29.349501869"
Nov 03 22:13:25.516041 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:25.515928    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:13:25.669662 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:25.669595    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:13:26.372973 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:26.372937    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-82h4r"
Nov 03 22:13:27.324402 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:27.324365    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:13:27.489797 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:27.489767    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:13:28.289519 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:28.289182    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1e33b576-1208-4325-87e9-4351920bfa06","Type":"ContainerStarted","Data":"fa2c478b6f6e2ff47de93363477c6d57a6bd6a5589bfc6ece133d3de475c38ed"}
Nov 03 22:13:28.290606 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:28.290083    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1e33b576-1208-4325-87e9-4351920bfa06","Type":"ContainerStarted","Data":"4bad046ecb8b5b1068e1ec5567a005daa35f9514a191799894a0fc284e418a69"}
Nov 03 22:13:28.293882 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:28.293844    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-gzwb4" event={"ID":"623e390b-6a39-48cd-9d7c-4a5890c11c6e","Type":"ContainerStarted","Data":"c5d7d2572757a308b6a54d07e3a1dbc3688c92561eae07588e4756459d09a038"}
Nov 03 22:13:28.294543 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:28.294515    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-gzwb4"
Nov 03 22:13:28.296118 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:28.295792    4431 patch_prober.go:29] interesting pod/downloads-6dd789c8b7-gzwb4 container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.133.0.11:8080/\": dial tcp 10.133.0.11:8080: connect: connection refused" start-of-body=
Nov 03 22:13:28.296118 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:28.295842    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-6dd789c8b7-gzwb4" podUID="623e390b-6a39-48cd-9d7c-4a5890c11c6e" containerName="download-server" probeResult="failure" output="Get \"http://10.133.0.11:8080/\": dial tcp 10.133.0.11:8080: connect: connection refused"
Nov 03 22:13:28.351249 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:28.351160    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-gzwb4" podStartSLOduration=1.649387748 podStartE2EDuration="19.351144954s" totalImagesPullingTime="17.701757206s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:13:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:13:28.34980319 +0000 UTC m=+32.375659053" watchObservedRunningTime="2026-11-03 22:13:28.351144954 +0000 UTC m=+32.377000816"
Nov 03 22:13:29.312497 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:29.311628    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-gzwb4"
Nov 03 22:13:29.923827 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:29.923787    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:13:30.091845 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:30.091761    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:13:31.260477 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:13:31.260447    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:13:31.313345 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:31.313318    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1e33b576-1208-4325-87e9-4351920bfa06","Type":"ContainerStarted","Data":"b9551a8a95f7a755cb0cf4649f97556cdc91383d0cf851b350d2035a1de54c83"}
Nov 03 22:13:31.313500 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:31.313372    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1e33b576-1208-4325-87e9-4351920bfa06","Type":"ContainerStarted","Data":"ffd11b079eac9d3158e351174756e300dd22ed9b441f9da11b0c3afded104a04"}
Nov 03 22:13:31.313500 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:31.313399    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1e33b576-1208-4325-87e9-4351920bfa06","Type":"ContainerStarted","Data":"4896d1d8515f07ea7f2021e198adc66c9be14bf35a69397d9bf02e31d8335cd8"}
Nov 03 22:13:31.413989 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:31.413900    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:13:31.561331 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:31.561303    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:13:31.698810 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:31.698725    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:13:32.186622 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:32.186590    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-fc6bcdddf-9zl84"
Nov 03 22:13:32.246307 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:32.246271    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:13:32.328735 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:32.328693    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1e33b576-1208-4325-87e9-4351920bfa06","Type":"ContainerStarted","Data":"bf6abfe3e4a0c89c18afb56290886e7a1de70da993232fa0350170875592b1f1"}
Nov 03 22:13:32.407670 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:32.407611    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.990552294 podStartE2EDuration="9.407589127s" totalImagesPullingTime="6.394537458s" totalInitContainerRuntime="22.499375ms" isStatefulPod=false podCreationTimestamp="2026-11-03 22:13:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:13:32.405997553 +0000 UTC m=+36.431853416" watchObservedRunningTime="2026-11-03 22:13:32.407589127 +0000 UTC m=+36.433444990"
Nov 03 22:13:32.414982 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:32.414945    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:13:32.574467 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:32.574375    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:13:32.739001 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:32.736128    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:13:33.680234 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:33.680175    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:13:35.402006 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:35.401968    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:13:35.541602 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:35.541568    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:13:38.201085 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:38.201052    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:13:38.341427 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:38.341397    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:13:39.695262 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:39.695204    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:13:39.736597 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:39.736571    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:13:40.374830 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:40.374795    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:13:43.869634 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:43.869585    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:13:44.034840 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:44.034793    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:13:47.234064 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:47.234023    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:13:47.376563 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:47.376530    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:13:52.508867 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:13:52.508832    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:14:01.258635 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:14:01.258598    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:14:02.171767 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:02.171725    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:14:02.309993 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:02.309958    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:14:05.487953 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:05.487918    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:14:05.635970 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:05.635927    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:14:05.777264 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:05.777147    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:14:05.917042 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:05.917007    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:14:07.258301 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:07.258260    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 22:14:07.258880 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:07.258769    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="43af3693-5cfe-4142-8880-b448d255c9a2" containerName="alertmanager" containerID="cri-o://3def159a0f1d846eb61c41295effcae60e18248b8a066e11d4edb17c38f0a6b6" gracePeriod=120
Nov 03 22:14:07.258880 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:07.258814    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="43af3693-5cfe-4142-8880-b448d255c9a2" containerName="prom-label-proxy" containerID="cri-o://21795324f5c89751e2ae8369fcd96bf0a88cc561a2a75b5163fac609ca6a06d2" gracePeriod=120
Nov 03 22:14:07.258880 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:07.258815    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="43af3693-5cfe-4142-8880-b448d255c9a2" containerName="kube-rbac-proxy-metric" containerID="cri-o://6e385664c9850bbc41e00b4bd353aeb782044a695cb066231e3222404dc8b745" gracePeriod=120
Nov 03 22:14:07.259047 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:07.258814    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="43af3693-5cfe-4142-8880-b448d255c9a2" containerName="kube-rbac-proxy" containerID="cri-o://da0d3be69e6ce965f95eb6dbb775cac4b1803b5f27909a36f1e67f6c9075de30" gracePeriod=120
Nov 03 22:14:07.259047 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:07.258836    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="43af3693-5cfe-4142-8880-b448d255c9a2" containerName="config-reloader" containerID="cri-o://6ae87ea73103189bd05e5802ae9a8ded55478d4333bf735c9eac3e0e87799436" gracePeriod=120
Nov 03 22:14:07.259047 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:07.258820    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="43af3693-5cfe-4142-8880-b448d255c9a2" containerName="kube-rbac-proxy-web" containerID="cri-o://a50202a2e2777b686132a5906736dce825d122c07868027079b183fac2359132" gracePeriod=120
Nov 03 22:14:07.457885 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:07.457853    4431 generic.go:356] "Generic (PLEG): container finished" podID="43af3693-5cfe-4142-8880-b448d255c9a2" containerID="21795324f5c89751e2ae8369fcd96bf0a88cc561a2a75b5163fac609ca6a06d2" exitCode=0
Nov 03 22:14:07.457885 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:07.457877    4431 generic.go:356] "Generic (PLEG): container finished" podID="43af3693-5cfe-4142-8880-b448d255c9a2" containerID="da0d3be69e6ce965f95eb6dbb775cac4b1803b5f27909a36f1e67f6c9075de30" exitCode=0
Nov 03 22:14:07.457885 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:07.457884    4431 generic.go:356] "Generic (PLEG): container finished" podID="43af3693-5cfe-4142-8880-b448d255c9a2" containerID="6ae87ea73103189bd05e5802ae9a8ded55478d4333bf735c9eac3e0e87799436" exitCode=0
Nov 03 22:14:07.457885 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:07.457892    4431 generic.go:356] "Generic (PLEG): container finished" podID="43af3693-5cfe-4142-8880-b448d255c9a2" containerID="3def159a0f1d846eb61c41295effcae60e18248b8a066e11d4edb17c38f0a6b6" exitCode=0
Nov 03 22:14:07.458116 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:07.457937    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"43af3693-5cfe-4142-8880-b448d255c9a2","Type":"ContainerDied","Data":"21795324f5c89751e2ae8369fcd96bf0a88cc561a2a75b5163fac609ca6a06d2"}
Nov 03 22:14:07.458116 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:07.458002    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"43af3693-5cfe-4142-8880-b448d255c9a2","Type":"ContainerDied","Data":"da0d3be69e6ce965f95eb6dbb775cac4b1803b5f27909a36f1e67f6c9075de30"}
Nov 03 22:14:07.458116 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:07.458028    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"43af3693-5cfe-4142-8880-b448d255c9a2","Type":"ContainerDied","Data":"6ae87ea73103189bd05e5802ae9a8ded55478d4333bf735c9eac3e0e87799436"}
Nov 03 22:14:07.458116 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:07.458051    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"43af3693-5cfe-4142-8880-b448d255c9a2","Type":"ContainerDied","Data":"3def159a0f1d846eb61c41295effcae60e18248b8a066e11d4edb17c38f0a6b6"}
Nov 03 22:14:08.467089 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:08.467059    4431 generic.go:356] "Generic (PLEG): container finished" podID="43af3693-5cfe-4142-8880-b448d255c9a2" containerID="6e385664c9850bbc41e00b4bd353aeb782044a695cb066231e3222404dc8b745" exitCode=0
Nov 03 22:14:08.467089 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:08.467085    4431 generic.go:356] "Generic (PLEG): container finished" podID="43af3693-5cfe-4142-8880-b448d255c9a2" containerID="a50202a2e2777b686132a5906736dce825d122c07868027079b183fac2359132" exitCode=0
Nov 03 22:14:08.467525 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:08.467133    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"43af3693-5cfe-4142-8880-b448d255c9a2","Type":"ContainerDied","Data":"6e385664c9850bbc41e00b4bd353aeb782044a695cb066231e3222404dc8b745"}
Nov 03 22:14:08.467525 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:08.467243    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"43af3693-5cfe-4142-8880-b448d255c9a2","Type":"ContainerDied","Data":"a50202a2e2777b686132a5906736dce825d122c07868027079b183fac2359132"}
Nov 03 22:14:08.524290 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:08.523413    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 22:14:08.527364 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:08.527338    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43af3693-5cfe-4142-8880-b448d255c9a2-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/43af3693-5cfe-4142-8880-b448d255c9a2-secret-alertmanager-kube-rbac-proxy-web\") pod \"43af3693-5cfe-4142-8880-b448d255c9a2\" (UID: \"43af3693-5cfe-4142-8880-b448d255c9a2\") "
Nov 03 22:14:08.527463 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:08.527389    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43af3693-5cfe-4142-8880-b448d255c9a2-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/43af3693-5cfe-4142-8880-b448d255c9a2-alertmanager-main-db\") pod \"43af3693-5cfe-4142-8880-b448d255c9a2\" (UID: \"43af3693-5cfe-4142-8880-b448d255c9a2\") "
Nov 03 22:14:08.527817 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:08.527426    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43af3693-5cfe-4142-8880-b448d255c9a2-web-config\" (UniqueName: \"kubernetes.io/secret/43af3693-5cfe-4142-8880-b448d255c9a2-web-config\") pod \"43af3693-5cfe-4142-8880-b448d255c9a2\" (UID: \"43af3693-5cfe-4142-8880-b448d255c9a2\") "
Nov 03 22:14:08.527914 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:08.527895    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43af3693-5cfe-4142-8880-b448d255c9a2-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/43af3693-5cfe-4142-8880-b448d255c9a2-secret-alertmanager-main-tls\") pod \"43af3693-5cfe-4142-8880-b448d255c9a2\" (UID: \"43af3693-5cfe-4142-8880-b448d255c9a2\") "
Nov 03 22:14:08.527960 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:08.527944    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43af3693-5cfe-4142-8880-b448d255c9a2-config-volume\" (UniqueName: \"kubernetes.io/secret/43af3693-5cfe-4142-8880-b448d255c9a2-config-volume\") pod \"43af3693-5cfe-4142-8880-b448d255c9a2\" (UID: \"43af3693-5cfe-4142-8880-b448d255c9a2\") "
Nov 03 22:14:08.528013 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:08.527979    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43af3693-5cfe-4142-8880-b448d255c9a2-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43af3693-5cfe-4142-8880-b448d255c9a2-alertmanager-trusted-ca-bundle\") pod \"43af3693-5cfe-4142-8880-b448d255c9a2\" (UID: \"43af3693-5cfe-4142-8880-b448d255c9a2\") "
Nov 03 22:14:08.528073 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:08.528010    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43af3693-5cfe-4142-8880-b448d255c9a2-config-out\" (UniqueName: \"kubernetes.io/empty-dir/43af3693-5cfe-4142-8880-b448d255c9a2-config-out\") pod \"43af3693-5cfe-4142-8880-b448d255c9a2\" (UID: \"43af3693-5cfe-4142-8880-b448d255c9a2\") "
Nov 03 22:14:08.528073 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:08.528042    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43af3693-5cfe-4142-8880-b448d255c9a2-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43af3693-5cfe-4142-8880-b448d255c9a2-metrics-client-ca\") pod \"43af3693-5cfe-4142-8880-b448d255c9a2\" (UID: \"43af3693-5cfe-4142-8880-b448d255c9a2\") "
Nov 03 22:14:08.528073 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:08.528056    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43af3693-5cfe-4142-8880-b448d255c9a2-alertmanager-main-db" pod "43af3693-5cfe-4142-8880-b448d255c9a2" (UID: "43af3693-5cfe-4142-8880-b448d255c9a2"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:14:08.528282 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:08.528077    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43af3693-5cfe-4142-8880-b448d255c9a2-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/43af3693-5cfe-4142-8880-b448d255c9a2-cluster-tls-config\") pod \"43af3693-5cfe-4142-8880-b448d255c9a2\" (UID: \"43af3693-5cfe-4142-8880-b448d255c9a2\") "
Nov 03 22:14:08.528282 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:08.528139    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43af3693-5cfe-4142-8880-b448d255c9a2-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/43af3693-5cfe-4142-8880-b448d255c9a2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"43af3693-5cfe-4142-8880-b448d255c9a2\" (UID: \"43af3693-5cfe-4142-8880-b448d255c9a2\") "
Nov 03 22:14:08.528282 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:08.528185    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43af3693-5cfe-4142-8880-b448d255c9a2-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/43af3693-5cfe-4142-8880-b448d255c9a2-secret-alertmanager-kube-rbac-proxy\") pod \"43af3693-5cfe-4142-8880-b448d255c9a2\" (UID: \"43af3693-5cfe-4142-8880-b448d255c9a2\") "
Nov 03 22:14:08.528282 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:08.528238    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43af3693-5cfe-4142-8880-b448d255c9a2-kube-api-access-q95nl\" (UniqueName: \"kubernetes.io/projected/43af3693-5cfe-4142-8880-b448d255c9a2-kube-api-access-q95nl\") pod \"43af3693-5cfe-4142-8880-b448d255c9a2\" (UID: \"43af3693-5cfe-4142-8880-b448d255c9a2\") "
Nov 03 22:14:08.528282 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:08.528263    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43af3693-5cfe-4142-8880-b448d255c9a2-tls-assets\" (UniqueName: \"kubernetes.io/projected/43af3693-5cfe-4142-8880-b448d255c9a2-tls-assets\") pod \"43af3693-5cfe-4142-8880-b448d255c9a2\" (UID: \"43af3693-5cfe-4142-8880-b448d255c9a2\") "
Nov 03 22:14:08.528535 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:08.528475    4431 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/43af3693-5cfe-4142-8880-b448d255c9a2-alertmanager-main-db\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:14:08.530934 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:08.530733    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43af3693-5cfe-4142-8880-b448d255c9a2-config-out" pod "43af3693-5cfe-4142-8880-b448d255c9a2" (UID: "43af3693-5cfe-4142-8880-b448d255c9a2"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:14:08.530934 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:08.530744    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43af3693-5cfe-4142-8880-b448d255c9a2-alertmanager-trusted-ca-bundle" pod "43af3693-5cfe-4142-8880-b448d255c9a2" (UID: "43af3693-5cfe-4142-8880-b448d255c9a2"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:14:08.531160 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:08.531134    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43af3693-5cfe-4142-8880-b448d255c9a2-config-volume" pod "43af3693-5cfe-4142-8880-b448d255c9a2" (UID: "43af3693-5cfe-4142-8880-b448d255c9a2"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:14:08.532870 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:08.532846    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43af3693-5cfe-4142-8880-b448d255c9a2-secret-alertmanager-main-tls" pod "43af3693-5cfe-4142-8880-b448d255c9a2" (UID: "43af3693-5cfe-4142-8880-b448d255c9a2"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:14:08.532967 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:08.532874    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43af3693-5cfe-4142-8880-b448d255c9a2-tls-assets" pod "43af3693-5cfe-4142-8880-b448d255c9a2" (UID: "43af3693-5cfe-4142-8880-b448d255c9a2"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:14:08.532967 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:08.532875    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43af3693-5cfe-4142-8880-b448d255c9a2-secret-alertmanager-kube-rbac-proxy-metric" pod "43af3693-5cfe-4142-8880-b448d255c9a2" (UID: "43af3693-5cfe-4142-8880-b448d255c9a2"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:14:08.532967 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:08.532935    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43af3693-5cfe-4142-8880-b448d255c9a2-secret-alertmanager-kube-rbac-proxy-web" pod "43af3693-5cfe-4142-8880-b448d255c9a2" (UID: "43af3693-5cfe-4142-8880-b448d255c9a2"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:14:08.533081 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:08.532977    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43af3693-5cfe-4142-8880-b448d255c9a2-metrics-client-ca" pod "43af3693-5cfe-4142-8880-b448d255c9a2" (UID: "43af3693-5cfe-4142-8880-b448d255c9a2"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:14:08.533081 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:08.532998    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43af3693-5cfe-4142-8880-b448d255c9a2-secret-alertmanager-kube-rbac-proxy" pod "43af3693-5cfe-4142-8880-b448d255c9a2" (UID: "43af3693-5cfe-4142-8880-b448d255c9a2"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:14:08.534399 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:08.534379    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43af3693-5cfe-4142-8880-b448d255c9a2-kube-api-access-q95nl" pod "43af3693-5cfe-4142-8880-b448d255c9a2" (UID: "43af3693-5cfe-4142-8880-b448d255c9a2"). InnerVolumeSpecName "kube-api-access-q95nl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:14:08.537799 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:08.537778    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43af3693-5cfe-4142-8880-b448d255c9a2-cluster-tls-config" pod "43af3693-5cfe-4142-8880-b448d255c9a2" (UID: "43af3693-5cfe-4142-8880-b448d255c9a2"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:14:08.545234 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:08.545200    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43af3693-5cfe-4142-8880-b448d255c9a2-web-config" pod "43af3693-5cfe-4142-8880-b448d255c9a2" (UID: "43af3693-5cfe-4142-8880-b448d255c9a2"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:14:08.629358 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:08.629076    4431 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/43af3693-5cfe-4142-8880-b448d255c9a2-tls-assets\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:14:08.629358 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:08.629114    4431 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/43af3693-5cfe-4142-8880-b448d255c9a2-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:14:08.629358 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:08.629130    4431 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/43af3693-5cfe-4142-8880-b448d255c9a2-web-config\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:14:08.629358 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:08.629144    4431 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/43af3693-5cfe-4142-8880-b448d255c9a2-secret-alertmanager-main-tls\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:14:08.629358 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:08.629160    4431 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/43af3693-5cfe-4142-8880-b448d255c9a2-config-volume\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:14:08.629358 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:08.629175    4431 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43af3693-5cfe-4142-8880-b448d255c9a2-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:14:08.629358 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:08.629189    4431 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/43af3693-5cfe-4142-8880-b448d255c9a2-config-out\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:14:08.629358 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:08.629204    4431 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43af3693-5cfe-4142-8880-b448d255c9a2-metrics-client-ca\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:14:08.629358 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:08.629241    4431 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/43af3693-5cfe-4142-8880-b448d255c9a2-cluster-tls-config\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:14:08.629358 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:08.629255    4431 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/43af3693-5cfe-4142-8880-b448d255c9a2-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:14:08.629358 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:08.629271    4431 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/43af3693-5cfe-4142-8880-b448d255c9a2-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:14:08.629358 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:08.629285    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q95nl\" (UniqueName: \"kubernetes.io/projected/43af3693-5cfe-4142-8880-b448d255c9a2-kube-api-access-q95nl\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:14:09.473364 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:09.473330    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"43af3693-5cfe-4142-8880-b448d255c9a2","Type":"ContainerDied","Data":"764eab8e9eee089c1ffbd8373e72f091286b43f93ce54f4b167ea338280b278b"}
Nov 03 22:14:09.473779 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:09.473376    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 22:14:09.473779 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:09.473400    4431 scope.go:122] "RemoveContainer" containerID="21795324f5c89751e2ae8369fcd96bf0a88cc561a2a75b5163fac609ca6a06d2"
Nov 03 22:14:09.481552 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:09.481530    4431 scope.go:122] "RemoveContainer" containerID="6e385664c9850bbc41e00b4bd353aeb782044a695cb066231e3222404dc8b745"
Nov 03 22:14:09.488759 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:09.488742    4431 scope.go:122] "RemoveContainer" containerID="da0d3be69e6ce965f95eb6dbb775cac4b1803b5f27909a36f1e67f6c9075de30"
Nov 03 22:14:09.495340 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:09.495322    4431 scope.go:122] "RemoveContainer" containerID="a50202a2e2777b686132a5906736dce825d122c07868027079b183fac2359132"
Nov 03 22:14:09.501773 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:09.501752    4431 scope.go:122] "RemoveContainer" containerID="6ae87ea73103189bd05e5802ae9a8ded55478d4333bf735c9eac3e0e87799436"
Nov 03 22:14:09.508446 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:09.508429    4431 scope.go:122] "RemoveContainer" containerID="3def159a0f1d846eb61c41295effcae60e18248b8a066e11d4edb17c38f0a6b6"
Nov 03 22:14:09.515082 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:09.515060    4431 scope.go:122] "RemoveContainer" containerID="4c45119c994eca23755699862d79d9b93fa7850e8018ec39c656087efe1fc13d"
Nov 03 22:14:09.540238 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:09.540193    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 22:14:09.553026 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:09.553004    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 22:14:09.578889 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:09.578863    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 22:14:09.579126 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:09.579115    4431 cpu_manager.go:414] "containerMap: removing container" podUID="43af3693-5cfe-4142-8880-b448d255c9a2" containerName="config-reloader"
Nov 03 22:14:09.579196 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:09.579128    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43af3693-5cfe-4142-8880-b448d255c9a2" containerName="config-reloader"
Nov 03 22:14:09.579196 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:09.579137    4431 cpu_manager.go:414] "containerMap: removing container" podUID="43af3693-5cfe-4142-8880-b448d255c9a2" containerName="kube-rbac-proxy-web"
Nov 03 22:14:09.579196 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:09.579142    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43af3693-5cfe-4142-8880-b448d255c9a2" containerName="kube-rbac-proxy-web"
Nov 03 22:14:09.579196 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:09.579154    4431 cpu_manager.go:414] "containerMap: removing container" podUID="43af3693-5cfe-4142-8880-b448d255c9a2" containerName="kube-rbac-proxy-metric"
Nov 03 22:14:09.579196 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:09.579159    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43af3693-5cfe-4142-8880-b448d255c9a2" containerName="kube-rbac-proxy-metric"
Nov 03 22:14:09.579196 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:09.579174    4431 cpu_manager.go:414] "containerMap: removing container" podUID="43af3693-5cfe-4142-8880-b448d255c9a2" containerName="kube-rbac-proxy"
Nov 03 22:14:09.579196 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:09.579178    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43af3693-5cfe-4142-8880-b448d255c9a2" containerName="kube-rbac-proxy"
Nov 03 22:14:09.579196 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:09.579186    4431 cpu_manager.go:414] "containerMap: removing container" podUID="43af3693-5cfe-4142-8880-b448d255c9a2" containerName="prom-label-proxy"
Nov 03 22:14:09.579196 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:09.579191    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43af3693-5cfe-4142-8880-b448d255c9a2" containerName="prom-label-proxy"
Nov 03 22:14:09.579456 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:09.579200    4431 cpu_manager.go:414] "containerMap: removing container" podUID="43af3693-5cfe-4142-8880-b448d255c9a2" containerName="init-config-reloader"
Nov 03 22:14:09.579456 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:09.579205    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43af3693-5cfe-4142-8880-b448d255c9a2" containerName="init-config-reloader"
Nov 03 22:14:09.579456 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:09.579225    4431 cpu_manager.go:414] "containerMap: removing container" podUID="43af3693-5cfe-4142-8880-b448d255c9a2" containerName="alertmanager"
Nov 03 22:14:09.579456 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:09.579230    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43af3693-5cfe-4142-8880-b448d255c9a2" containerName="alertmanager"
Nov 03 22:14:09.579456 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:09.579279    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="43af3693-5cfe-4142-8880-b448d255c9a2" containerName="config-reloader"
Nov 03 22:14:09.579456 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:09.579287    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="43af3693-5cfe-4142-8880-b448d255c9a2" containerName="kube-rbac-proxy-web"
Nov 03 22:14:09.579456 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:09.579294    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="43af3693-5cfe-4142-8880-b448d255c9a2" containerName="kube-rbac-proxy-metric"
Nov 03 22:14:09.579456 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:09.579301    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="43af3693-5cfe-4142-8880-b448d255c9a2" containerName="kube-rbac-proxy"
Nov 03 22:14:09.579456 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:09.579308    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="43af3693-5cfe-4142-8880-b448d255c9a2" containerName="prom-label-proxy"
Nov 03 22:14:09.579456 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:09.579314    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="43af3693-5cfe-4142-8880-b448d255c9a2" containerName="alertmanager"
Nov 03 22:14:09.613627 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:09.613572    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 22:14:09.632449 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:09.632415    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 03 22:14:09.634620 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:09.634589    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 22:14:09.636350 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:09.636314    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 03 22:14:09.636492 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:09.636469    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 03 22:14:09.636610 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:09.636554    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 03 22:14:09.636610 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:09.636573    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/797dcec6-675d-4af2-89a4-1b5902ae547a-config-out\") pod \"alertmanager-main-0\" (UID: \"797dcec6-675d-4af2-89a4-1b5902ae547a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 22:14:09.636728 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:09.636615    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/797dcec6-675d-4af2-89a4-1b5902ae547a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"797dcec6-675d-4af2-89a4-1b5902ae547a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 22:14:09.636728 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:09.636648    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/797dcec6-675d-4af2-89a4-1b5902ae547a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"797dcec6-675d-4af2-89a4-1b5902ae547a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 22:14:09.636813 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:09.636726    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/797dcec6-675d-4af2-89a4-1b5902ae547a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"797dcec6-675d-4af2-89a4-1b5902ae547a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 22:14:09.636813 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:09.636796    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/797dcec6-675d-4af2-89a4-1b5902ae547a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"797dcec6-675d-4af2-89a4-1b5902ae547a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 22:14:09.636911 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:09.636830    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/797dcec6-675d-4af2-89a4-1b5902ae547a-web-config\") pod \"alertmanager-main-0\" (UID: \"797dcec6-675d-4af2-89a4-1b5902ae547a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 22:14:09.636911 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:09.636890    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/797dcec6-675d-4af2-89a4-1b5902ae547a-config-volume\") pod \"alertmanager-main-0\" (UID: \"797dcec6-675d-4af2-89a4-1b5902ae547a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 22:14:09.636972 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:09.636944    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2wjt\" (UniqueName: \"kubernetes.io/projected/797dcec6-675d-4af2-89a4-1b5902ae547a-kube-api-access-t2wjt\") pod \"alertmanager-main-0\" (UID: \"797dcec6-675d-4af2-89a4-1b5902ae547a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 22:14:09.637046 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:09.637031    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/797dcec6-675d-4af2-89a4-1b5902ae547a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"797dcec6-675d-4af2-89a4-1b5902ae547a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 22:14:09.637096 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:09.637055    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/797dcec6-675d-4af2-89a4-1b5902ae547a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"797dcec6-675d-4af2-89a4-1b5902ae547a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 22:14:09.637157 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:09.637136    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/797dcec6-675d-4af2-89a4-1b5902ae547a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"797dcec6-675d-4af2-89a4-1b5902ae547a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 22:14:09.637245 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:09.637209    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/797dcec6-675d-4af2-89a4-1b5902ae547a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"797dcec6-675d-4af2-89a4-1b5902ae547a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 22:14:09.637307 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:09.637285    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/797dcec6-675d-4af2-89a4-1b5902ae547a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"797dcec6-675d-4af2-89a4-1b5902ae547a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 22:14:09.637419 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:09.637400    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 03 22:14:09.650072 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:09.650050    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 03 22:14:09.731944 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:09.731911    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 03 22:14:09.732125 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:09.731992    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 03 22:14:09.732125 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:09.732122    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-pw7hg\""
Nov 03 22:14:09.732293 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:09.732127    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 03 22:14:09.737899 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:09.737878    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/797dcec6-675d-4af2-89a4-1b5902ae547a-config-out\") pod \"alertmanager-main-0\" (UID: \"797dcec6-675d-4af2-89a4-1b5902ae547a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 22:14:09.737965 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:09.737909    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/797dcec6-675d-4af2-89a4-1b5902ae547a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"797dcec6-675d-4af2-89a4-1b5902ae547a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 22:14:09.737965 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:09.737928    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/797dcec6-675d-4af2-89a4-1b5902ae547a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"797dcec6-675d-4af2-89a4-1b5902ae547a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 22:14:09.737965 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:09.737943    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/797dcec6-675d-4af2-89a4-1b5902ae547a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"797dcec6-675d-4af2-89a4-1b5902ae547a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 22:14:09.737965 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:09.737963    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/797dcec6-675d-4af2-89a4-1b5902ae547a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"797dcec6-675d-4af2-89a4-1b5902ae547a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 22:14:09.738157 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:09.737987    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/797dcec6-675d-4af2-89a4-1b5902ae547a-web-config\") pod \"alertmanager-main-0\" (UID: \"797dcec6-675d-4af2-89a4-1b5902ae547a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 22:14:09.738157 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:09.738010    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/797dcec6-675d-4af2-89a4-1b5902ae547a-config-volume\") pod \"alertmanager-main-0\" (UID: \"797dcec6-675d-4af2-89a4-1b5902ae547a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 22:14:09.738157 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:09.738133    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2wjt\" (UniqueName: \"kubernetes.io/projected/797dcec6-675d-4af2-89a4-1b5902ae547a-kube-api-access-t2wjt\") pod \"alertmanager-main-0\" (UID: \"797dcec6-675d-4af2-89a4-1b5902ae547a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 22:14:09.738325 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:09.738187    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/797dcec6-675d-4af2-89a4-1b5902ae547a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"797dcec6-675d-4af2-89a4-1b5902ae547a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 22:14:09.738325 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:09.738244    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/797dcec6-675d-4af2-89a4-1b5902ae547a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"797dcec6-675d-4af2-89a4-1b5902ae547a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 22:14:09.738434 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:09.738336    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/797dcec6-675d-4af2-89a4-1b5902ae547a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"797dcec6-675d-4af2-89a4-1b5902ae547a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 22:14:09.738434 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:09.738365    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/797dcec6-675d-4af2-89a4-1b5902ae547a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"797dcec6-675d-4af2-89a4-1b5902ae547a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 22:14:09.738434 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:09.738422    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/797dcec6-675d-4af2-89a4-1b5902ae547a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"797dcec6-675d-4af2-89a4-1b5902ae547a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 22:14:09.738594 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:09.738423    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/797dcec6-675d-4af2-89a4-1b5902ae547a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"797dcec6-675d-4af2-89a4-1b5902ae547a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 22:14:09.738754 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:09.738725    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/797dcec6-675d-4af2-89a4-1b5902ae547a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"797dcec6-675d-4af2-89a4-1b5902ae547a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 22:14:09.738937 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:09.738916    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/797dcec6-675d-4af2-89a4-1b5902ae547a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"797dcec6-675d-4af2-89a4-1b5902ae547a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 22:14:09.741579 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:09.741403    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/797dcec6-675d-4af2-89a4-1b5902ae547a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"797dcec6-675d-4af2-89a4-1b5902ae547a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 22:14:09.741579 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:09.741515    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/797dcec6-675d-4af2-89a4-1b5902ae547a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"797dcec6-675d-4af2-89a4-1b5902ae547a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 22:14:09.741579 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:09.741548    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/797dcec6-675d-4af2-89a4-1b5902ae547a-config-out\") pod \"alertmanager-main-0\" (UID: \"797dcec6-675d-4af2-89a4-1b5902ae547a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 22:14:09.741794 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:09.741742    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/797dcec6-675d-4af2-89a4-1b5902ae547a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"797dcec6-675d-4af2-89a4-1b5902ae547a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 22:14:09.741847 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:09.741802    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/797dcec6-675d-4af2-89a4-1b5902ae547a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"797dcec6-675d-4af2-89a4-1b5902ae547a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 22:14:09.741847 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:09.741831    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/797dcec6-675d-4af2-89a4-1b5902ae547a-web-config\") pod \"alertmanager-main-0\" (UID: \"797dcec6-675d-4af2-89a4-1b5902ae547a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 22:14:09.741946 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:09.741872    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/797dcec6-675d-4af2-89a4-1b5902ae547a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"797dcec6-675d-4af2-89a4-1b5902ae547a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 22:14:09.742300 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:09.742282    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/797dcec6-675d-4af2-89a4-1b5902ae547a-config-volume\") pod \"alertmanager-main-0\" (UID: \"797dcec6-675d-4af2-89a4-1b5902ae547a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 22:14:09.742935 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:09.742919    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/797dcec6-675d-4af2-89a4-1b5902ae547a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"797dcec6-675d-4af2-89a4-1b5902ae547a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 22:14:09.765752 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:09.765724    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2wjt\" (UniqueName: \"kubernetes.io/projected/797dcec6-675d-4af2-89a4-1b5902ae547a-kube-api-access-t2wjt\") pod \"alertmanager-main-0\" (UID: \"797dcec6-675d-4af2-89a4-1b5902ae547a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 22:14:10.004010 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:10.003979    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:14:10.013839 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:10.013815    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 22:14:10.085611 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:10.085579    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43af3693-5cfe-4142-8880-b448d255c9a2" path="/var/lib/kubelet/pods/43af3693-5cfe-4142-8880-b448d255c9a2/volumes"
Nov 03 22:14:10.144933 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:10.144897    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:14:10.374567 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:10.374486    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 22:14:10.389510 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:14:10.389481    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod797dcec6_675d_4af2_89a4_1b5902ae547a.slice/crio-62d4b5617555e50b3fec8a9ad0cf5ef982d52b8f56218e33149ea59a775ab3c9 WatchSource:0}: Error finding container 62d4b5617555e50b3fec8a9ad0cf5ef982d52b8f56218e33149ea59a775ab3c9: Status 404 returned error can't find the container with id 62d4b5617555e50b3fec8a9ad0cf5ef982d52b8f56218e33149ea59a775ab3c9
Nov 03 22:14:10.479748 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:10.479717    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"797dcec6-675d-4af2-89a4-1b5902ae547a","Type":"ContainerStarted","Data":"62d4b5617555e50b3fec8a9ad0cf5ef982d52b8f56218e33149ea59a775ab3c9"}
Nov 03 22:14:11.486508 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:11.486473    4431 generic.go:356] "Generic (PLEG): container finished" podID="797dcec6-675d-4af2-89a4-1b5902ae547a" containerID="be9cd93cc2e634b0aab1ebbdceece9b9eed0c4401fb6bb450310b82d052ed0d1" exitCode=0
Nov 03 22:14:11.486873 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:11.486558    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"797dcec6-675d-4af2-89a4-1b5902ae547a","Type":"ContainerDied","Data":"be9cd93cc2e634b0aab1ebbdceece9b9eed0c4401fb6bb450310b82d052ed0d1"}
Nov 03 22:14:11.874315 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:11.874291    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 22:14:11.874587 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:11.874561    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1e33b576-1208-4325-87e9-4351920bfa06" containerName="prometheus" containerID="cri-o://4bad046ecb8b5b1068e1ec5567a005daa35f9514a191799894a0fc284e418a69" gracePeriod=600
Nov 03 22:14:11.874892 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:11.874870    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1e33b576-1208-4325-87e9-4351920bfa06" containerName="kube-rbac-proxy-thanos" containerID="cri-o://bf6abfe3e4a0c89c18afb56290886e7a1de70da993232fa0350170875592b1f1" gracePeriod=600
Nov 03 22:14:11.874966 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:11.874953    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1e33b576-1208-4325-87e9-4351920bfa06" containerName="kube-rbac-proxy" containerID="cri-o://b9551a8a95f7a755cb0cf4649f97556cdc91383d0cf851b350d2035a1de54c83" gracePeriod=600
Nov 03 22:14:11.875021 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:11.875006    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1e33b576-1208-4325-87e9-4351920bfa06" containerName="kube-rbac-proxy-web" containerID="cri-o://ffd11b079eac9d3158e351174756e300dd22ed9b441f9da11b0c3afded104a04" gracePeriod=600
Nov 03 22:14:11.875084 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:11.875057    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1e33b576-1208-4325-87e9-4351920bfa06" containerName="thanos-sidecar" containerID="cri-o://4896d1d8515f07ea7f2021e198adc66c9be14bf35a69397d9bf02e31d8335cd8" gracePeriod=600
Nov 03 22:14:11.875130 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:11.875117    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1e33b576-1208-4325-87e9-4351920bfa06" containerName="config-reloader" containerID="cri-o://fa2c478b6f6e2ff47de93363477c6d57a6bd6a5589bfc6ece133d3de475c38ed" gracePeriod=600
Nov 03 22:14:12.172098 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.172065    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:14:12.256629 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.256598    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e33b576-1208-4325-87e9-4351920bfa06-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1e33b576-1208-4325-87e9-4351920bfa06-secret-grpc-tls\") pod \"1e33b576-1208-4325-87e9-4351920bfa06\" (UID: \"1e33b576-1208-4325-87e9-4351920bfa06\") "
Nov 03 22:14:12.256792 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.256640    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e33b576-1208-4325-87e9-4351920bfa06-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1e33b576-1208-4325-87e9-4351920bfa06-thanos-prometheus-http-client-file\") pod \"1e33b576-1208-4325-87e9-4351920bfa06\" (UID: \"1e33b576-1208-4325-87e9-4351920bfa06\") "
Nov 03 22:14:12.256792 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.256685    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e33b576-1208-4325-87e9-4351920bfa06-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1e33b576-1208-4325-87e9-4351920bfa06-prometheus-k8s-db\") pod \"1e33b576-1208-4325-87e9-4351920bfa06\" (UID: \"1e33b576-1208-4325-87e9-4351920bfa06\") "
Nov 03 22:14:12.256792 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.256707    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e33b576-1208-4325-87e9-4351920bfa06-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1e33b576-1208-4325-87e9-4351920bfa06-prometheus-k8s-rulefiles-0\") pod \"1e33b576-1208-4325-87e9-4351920bfa06\" (UID: \"1e33b576-1208-4325-87e9-4351920bfa06\") "
Nov 03 22:14:12.256792 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.256733    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e33b576-1208-4325-87e9-4351920bfa06-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1e33b576-1208-4325-87e9-4351920bfa06-secret-metrics-client-certs\") pod \"1e33b576-1208-4325-87e9-4351920bfa06\" (UID: \"1e33b576-1208-4325-87e9-4351920bfa06\") "
Nov 03 22:14:12.256792 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.256764    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e33b576-1208-4325-87e9-4351920bfa06-config-out\" (UniqueName: \"kubernetes.io/empty-dir/1e33b576-1208-4325-87e9-4351920bfa06-config-out\") pod \"1e33b576-1208-4325-87e9-4351920bfa06\" (UID: \"1e33b576-1208-4325-87e9-4351920bfa06\") "
Nov 03 22:14:12.257040 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.256795    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e33b576-1208-4325-87e9-4351920bfa06-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e33b576-1208-4325-87e9-4351920bfa06-prometheus-trusted-ca-bundle\") pod \"1e33b576-1208-4325-87e9-4351920bfa06\" (UID: \"1e33b576-1208-4325-87e9-4351920bfa06\") "
Nov 03 22:14:12.257040 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.256832    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e33b576-1208-4325-87e9-4351920bfa06-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1e33b576-1208-4325-87e9-4351920bfa06-configmap-metrics-client-ca\") pod \"1e33b576-1208-4325-87e9-4351920bfa06\" (UID: \"1e33b576-1208-4325-87e9-4351920bfa06\") "
Nov 03 22:14:12.257040 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.256868    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e33b576-1208-4325-87e9-4351920bfa06-web-config\" (UniqueName: \"kubernetes.io/secret/1e33b576-1208-4325-87e9-4351920bfa06-web-config\") pod \"1e33b576-1208-4325-87e9-4351920bfa06\" (UID: \"1e33b576-1208-4325-87e9-4351920bfa06\") "
Nov 03 22:14:12.257040 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.256902    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e33b576-1208-4325-87e9-4351920bfa06-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1e33b576-1208-4325-87e9-4351920bfa06-secret-kube-rbac-proxy\") pod \"1e33b576-1208-4325-87e9-4351920bfa06\" (UID: \"1e33b576-1208-4325-87e9-4351920bfa06\") "
Nov 03 22:14:12.257040 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.256933    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e33b576-1208-4325-87e9-4351920bfa06-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1e33b576-1208-4325-87e9-4351920bfa06-secret-prometheus-k8s-tls\") pod \"1e33b576-1208-4325-87e9-4351920bfa06\" (UID: \"1e33b576-1208-4325-87e9-4351920bfa06\") "
Nov 03 22:14:12.257040 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.256960    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e33b576-1208-4325-87e9-4351920bfa06-kube-api-access-sxpzp\" (UniqueName: \"kubernetes.io/projected/1e33b576-1208-4325-87e9-4351920bfa06-kube-api-access-sxpzp\") pod \"1e33b576-1208-4325-87e9-4351920bfa06\" (UID: \"1e33b576-1208-4325-87e9-4351920bfa06\") "
Nov 03 22:14:12.257040 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.256994    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e33b576-1208-4325-87e9-4351920bfa06-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e33b576-1208-4325-87e9-4351920bfa06-configmap-serving-certs-ca-bundle\") pod \"1e33b576-1208-4325-87e9-4351920bfa06\" (UID: \"1e33b576-1208-4325-87e9-4351920bfa06\") "
Nov 03 22:14:12.257040 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.257022    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e33b576-1208-4325-87e9-4351920bfa06-tls-assets\" (UniqueName: \"kubernetes.io/projected/1e33b576-1208-4325-87e9-4351920bfa06-tls-assets\") pod \"1e33b576-1208-4325-87e9-4351920bfa06\" (UID: \"1e33b576-1208-4325-87e9-4351920bfa06\") "
Nov 03 22:14:12.257454 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.257057    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e33b576-1208-4325-87e9-4351920bfa06-config\" (UniqueName: \"kubernetes.io/secret/1e33b576-1208-4325-87e9-4351920bfa06-config\") pod \"1e33b576-1208-4325-87e9-4351920bfa06\" (UID: \"1e33b576-1208-4325-87e9-4351920bfa06\") "
Nov 03 22:14:12.257454 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.257086    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e33b576-1208-4325-87e9-4351920bfa06-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e33b576-1208-4325-87e9-4351920bfa06-configmap-kubelet-serving-ca-bundle\") pod \"1e33b576-1208-4325-87e9-4351920bfa06\" (UID: \"1e33b576-1208-4325-87e9-4351920bfa06\") "
Nov 03 22:14:12.257454 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.257139    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e33b576-1208-4325-87e9-4351920bfa06-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1e33b576-1208-4325-87e9-4351920bfa06-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"1e33b576-1208-4325-87e9-4351920bfa06\" (UID: \"1e33b576-1208-4325-87e9-4351920bfa06\") "
Nov 03 22:14:12.257454 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.257168    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e33b576-1208-4325-87e9-4351920bfa06-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1e33b576-1208-4325-87e9-4351920bfa06-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"1e33b576-1208-4325-87e9-4351920bfa06\" (UID: \"1e33b576-1208-4325-87e9-4351920bfa06\") "
Nov 03 22:14:12.257454 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.257196    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e33b576-1208-4325-87e9-4351920bfa06-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1e33b576-1208-4325-87e9-4351920bfa06-prometheus-k8s-rulefiles-1\") pod \"1e33b576-1208-4325-87e9-4351920bfa06\" (UID: \"1e33b576-1208-4325-87e9-4351920bfa06\") "
Nov 03 22:14:12.257454 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.257242    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e33b576-1208-4325-87e9-4351920bfa06-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1e33b576-1208-4325-87e9-4351920bfa06-prometheus-k8s-rulefiles-2\") pod \"1e33b576-1208-4325-87e9-4351920bfa06\" (UID: \"1e33b576-1208-4325-87e9-4351920bfa06\") "
Nov 03 22:14:12.257454 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.257310    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e33b576-1208-4325-87e9-4351920bfa06-prometheus-trusted-ca-bundle" pod "1e33b576-1208-4325-87e9-4351920bfa06" (UID: "1e33b576-1208-4325-87e9-4351920bfa06"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:14:12.257785 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.257543    4431 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e33b576-1208-4325-87e9-4351920bfa06-prometheus-trusted-ca-bundle\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:14:12.257785 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.257691    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e33b576-1208-4325-87e9-4351920bfa06-configmap-metrics-client-ca" pod "1e33b576-1208-4325-87e9-4351920bfa06" (UID: "1e33b576-1208-4325-87e9-4351920bfa06"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:14:12.257785 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.257705    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e33b576-1208-4325-87e9-4351920bfa06-prometheus-k8s-rulefiles-2" pod "1e33b576-1208-4325-87e9-4351920bfa06" (UID: "1e33b576-1208-4325-87e9-4351920bfa06"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:14:12.258497 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.258048    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e33b576-1208-4325-87e9-4351920bfa06-prometheus-k8s-db" pod "1e33b576-1208-4325-87e9-4351920bfa06" (UID: "1e33b576-1208-4325-87e9-4351920bfa06"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:14:12.258497 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.258075    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e33b576-1208-4325-87e9-4351920bfa06-configmap-serving-certs-ca-bundle" pod "1e33b576-1208-4325-87e9-4351920bfa06" (UID: "1e33b576-1208-4325-87e9-4351920bfa06"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:14:12.258497 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.258421    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e33b576-1208-4325-87e9-4351920bfa06-configmap-kubelet-serving-ca-bundle" pod "1e33b576-1208-4325-87e9-4351920bfa06" (UID: "1e33b576-1208-4325-87e9-4351920bfa06"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:14:12.258731 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.258627    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e33b576-1208-4325-87e9-4351920bfa06-prometheus-k8s-rulefiles-0" pod "1e33b576-1208-4325-87e9-4351920bfa06" (UID: "1e33b576-1208-4325-87e9-4351920bfa06"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:14:12.259893 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.259853    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e33b576-1208-4325-87e9-4351920bfa06-prometheus-k8s-rulefiles-1" pod "1e33b576-1208-4325-87e9-4351920bfa06" (UID: "1e33b576-1208-4325-87e9-4351920bfa06"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:14:12.260362 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.260307    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e33b576-1208-4325-87e9-4351920bfa06-secret-metrics-client-certs" pod "1e33b576-1208-4325-87e9-4351920bfa06" (UID: "1e33b576-1208-4325-87e9-4351920bfa06"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:14:12.260859 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.260837    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e33b576-1208-4325-87e9-4351920bfa06-tls-assets" pod "1e33b576-1208-4325-87e9-4351920bfa06" (UID: "1e33b576-1208-4325-87e9-4351920bfa06"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:14:12.261778 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.261752    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e33b576-1208-4325-87e9-4351920bfa06-secret-prometheus-k8s-thanos-sidecar-tls" pod "1e33b576-1208-4325-87e9-4351920bfa06" (UID: "1e33b576-1208-4325-87e9-4351920bfa06"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:14:12.261778 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.261753    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e33b576-1208-4325-87e9-4351920bfa06-kube-api-access-sxpzp" pod "1e33b576-1208-4325-87e9-4351920bfa06" (UID: "1e33b576-1208-4325-87e9-4351920bfa06"). InnerVolumeSpecName "kube-api-access-sxpzp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:14:12.261937 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.261794    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e33b576-1208-4325-87e9-4351920bfa06-secret-kube-rbac-proxy" pod "1e33b576-1208-4325-87e9-4351920bfa06" (UID: "1e33b576-1208-4325-87e9-4351920bfa06"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:14:12.261937 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.261855    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e33b576-1208-4325-87e9-4351920bfa06-secret-grpc-tls" pod "1e33b576-1208-4325-87e9-4351920bfa06" (UID: "1e33b576-1208-4325-87e9-4351920bfa06"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:14:12.262623 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.262587    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e33b576-1208-4325-87e9-4351920bfa06-config" pod "1e33b576-1208-4325-87e9-4351920bfa06" (UID: "1e33b576-1208-4325-87e9-4351920bfa06"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:14:12.262725 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.262626    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e33b576-1208-4325-87e9-4351920bfa06-secret-prometheus-k8s-tls" pod "1e33b576-1208-4325-87e9-4351920bfa06" (UID: "1e33b576-1208-4325-87e9-4351920bfa06"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:14:12.262725 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.262695    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e33b576-1208-4325-87e9-4351920bfa06-thanos-prometheus-http-client-file" pod "1e33b576-1208-4325-87e9-4351920bfa06" (UID: "1e33b576-1208-4325-87e9-4351920bfa06"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:14:12.263269 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.263251    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e33b576-1208-4325-87e9-4351920bfa06-secret-prometheus-k8s-kube-rbac-proxy-web" pod "1e33b576-1208-4325-87e9-4351920bfa06" (UID: "1e33b576-1208-4325-87e9-4351920bfa06"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:14:12.263636 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.263617    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e33b576-1208-4325-87e9-4351920bfa06-config-out" pod "1e33b576-1208-4325-87e9-4351920bfa06" (UID: "1e33b576-1208-4325-87e9-4351920bfa06"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:14:12.273675 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.273648    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e33b576-1208-4325-87e9-4351920bfa06-web-config" pod "1e33b576-1208-4325-87e9-4351920bfa06" (UID: "1e33b576-1208-4325-87e9-4351920bfa06"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:14:12.357924 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.357880    4431 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1e33b576-1208-4325-87e9-4351920bfa06-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:14:12.357924 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.357922    4431 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1e33b576-1208-4325-87e9-4351920bfa06-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:14:12.358106 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.357933    4431 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1e33b576-1208-4325-87e9-4351920bfa06-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:14:12.358106 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.357942    4431 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1e33b576-1208-4325-87e9-4351920bfa06-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:14:12.358106 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.357951    4431 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1e33b576-1208-4325-87e9-4351920bfa06-secret-grpc-tls\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:14:12.358106 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.357961    4431 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1e33b576-1208-4325-87e9-4351920bfa06-thanos-prometheus-http-client-file\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:14:12.358106 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.357971    4431 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1e33b576-1208-4325-87e9-4351920bfa06-prometheus-k8s-db\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:14:12.358106 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.357979    4431 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1e33b576-1208-4325-87e9-4351920bfa06-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:14:12.358106 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.357988    4431 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1e33b576-1208-4325-87e9-4351920bfa06-secret-metrics-client-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:14:12.358106 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.357998    4431 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1e33b576-1208-4325-87e9-4351920bfa06-config-out\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:14:12.358106 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.358006    4431 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1e33b576-1208-4325-87e9-4351920bfa06-configmap-metrics-client-ca\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:14:12.358106 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.358014    4431 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1e33b576-1208-4325-87e9-4351920bfa06-web-config\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:14:12.358106 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.358024    4431 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1e33b576-1208-4325-87e9-4351920bfa06-secret-kube-rbac-proxy\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:14:12.358106 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.358033    4431 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1e33b576-1208-4325-87e9-4351920bfa06-secret-prometheus-k8s-tls\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:14:12.358106 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.358043    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxpzp\" (UniqueName: \"kubernetes.io/projected/1e33b576-1208-4325-87e9-4351920bfa06-kube-api-access-sxpzp\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:14:12.358106 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.358051    4431 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e33b576-1208-4325-87e9-4351920bfa06-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:14:12.358106 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.358059    4431 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1e33b576-1208-4325-87e9-4351920bfa06-tls-assets\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:14:12.358106 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.358067    4431 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/1e33b576-1208-4325-87e9-4351920bfa06-config\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:14:12.358106 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.358076    4431 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e33b576-1208-4325-87e9-4351920bfa06-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:14:12.495832 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.495794    4431 generic.go:356] "Generic (PLEG): container finished" podID="1e33b576-1208-4325-87e9-4351920bfa06" containerID="bf6abfe3e4a0c89c18afb56290886e7a1de70da993232fa0350170875592b1f1" exitCode=0
Nov 03 22:14:12.495832 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.495825    4431 generic.go:356] "Generic (PLEG): container finished" podID="1e33b576-1208-4325-87e9-4351920bfa06" containerID="b9551a8a95f7a755cb0cf4649f97556cdc91383d0cf851b350d2035a1de54c83" exitCode=0
Nov 03 22:14:12.495832 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.495833    4431 generic.go:356] "Generic (PLEG): container finished" podID="1e33b576-1208-4325-87e9-4351920bfa06" containerID="ffd11b079eac9d3158e351174756e300dd22ed9b441f9da11b0c3afded104a04" exitCode=0
Nov 03 22:14:12.496422 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.495842    4431 generic.go:356] "Generic (PLEG): container finished" podID="1e33b576-1208-4325-87e9-4351920bfa06" containerID="4896d1d8515f07ea7f2021e198adc66c9be14bf35a69397d9bf02e31d8335cd8" exitCode=0
Nov 03 22:14:12.496422 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.495851    4431 generic.go:356] "Generic (PLEG): container finished" podID="1e33b576-1208-4325-87e9-4351920bfa06" containerID="fa2c478b6f6e2ff47de93363477c6d57a6bd6a5589bfc6ece133d3de475c38ed" exitCode=0
Nov 03 22:14:12.496422 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.495859    4431 generic.go:356] "Generic (PLEG): container finished" podID="1e33b576-1208-4325-87e9-4351920bfa06" containerID="4bad046ecb8b5b1068e1ec5567a005daa35f9514a191799894a0fc284e418a69" exitCode=0
Nov 03 22:14:12.496422 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.495875    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1e33b576-1208-4325-87e9-4351920bfa06","Type":"ContainerDied","Data":"bf6abfe3e4a0c89c18afb56290886e7a1de70da993232fa0350170875592b1f1"}
Nov 03 22:14:12.496422 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.495889    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:14:12.496422 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.495938    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1e33b576-1208-4325-87e9-4351920bfa06","Type":"ContainerDied","Data":"b9551a8a95f7a755cb0cf4649f97556cdc91383d0cf851b350d2035a1de54c83"}
Nov 03 22:14:12.496422 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.495976    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1e33b576-1208-4325-87e9-4351920bfa06","Type":"ContainerDied","Data":"ffd11b079eac9d3158e351174756e300dd22ed9b441f9da11b0c3afded104a04"}
Nov 03 22:14:12.496422 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.496002    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1e33b576-1208-4325-87e9-4351920bfa06","Type":"ContainerDied","Data":"4896d1d8515f07ea7f2021e198adc66c9be14bf35a69397d9bf02e31d8335cd8"}
Nov 03 22:14:12.496422 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.496026    4431 scope.go:122] "RemoveContainer" containerID="bf6abfe3e4a0c89c18afb56290886e7a1de70da993232fa0350170875592b1f1"
Nov 03 22:14:12.496422 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.496040    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1e33b576-1208-4325-87e9-4351920bfa06","Type":"ContainerDied","Data":"fa2c478b6f6e2ff47de93363477c6d57a6bd6a5589bfc6ece133d3de475c38ed"}
Nov 03 22:14:12.496422 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.496080    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1e33b576-1208-4325-87e9-4351920bfa06","Type":"ContainerDied","Data":"4bad046ecb8b5b1068e1ec5567a005daa35f9514a191799894a0fc284e418a69"}
Nov 03 22:14:12.496422 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.496125    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1e33b576-1208-4325-87e9-4351920bfa06","Type":"ContainerDied","Data":"3ee14ca7ee632401e828940c6d8369458df64a5342e922765e2814e1d62f253d"}
Nov 03 22:14:12.499627 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.499602    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"797dcec6-675d-4af2-89a4-1b5902ae547a","Type":"ContainerStarted","Data":"3b1beb32cd6a7057a22e6d342f6bb46e094613b0d5c68f3fdb6a6e3fa6298d76"}
Nov 03 22:14:12.499717 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.499674    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"797dcec6-675d-4af2-89a4-1b5902ae547a","Type":"ContainerStarted","Data":"7276a1efce871ca39539f874f9d6cc21e5f5dfa01abc9a16f7f9b0b835de89c2"}
Nov 03 22:14:12.499759 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.499719    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"797dcec6-675d-4af2-89a4-1b5902ae547a","Type":"ContainerStarted","Data":"659b60fa50952f0803665750fe3b01e7dc23be008f983aa7ef89786d47fa3eca"}
Nov 03 22:14:12.499759 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.499747    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"797dcec6-675d-4af2-89a4-1b5902ae547a","Type":"ContainerStarted","Data":"b9cf5ce91fbf5e0fe495b3fff2f4bac1ba29eef61085ba201fb5cf87bb20080b"}
Nov 03 22:14:12.499820 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.499771    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"797dcec6-675d-4af2-89a4-1b5902ae547a","Type":"ContainerStarted","Data":"9210fcc30dd7b2a23201526ad4f171fdf01dd5480bf73a350eb30aa4afb648e6"}
Nov 03 22:14:12.499820 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.499810    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"797dcec6-675d-4af2-89a4-1b5902ae547a","Type":"ContainerStarted","Data":"ae8d6260cbcd5d6e787a9880a13621184138f4b442b7a80634e821b197828ff8"}
Nov 03 22:14:12.504163 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.504106    4431 scope.go:122] "RemoveContainer" containerID="b9551a8a95f7a755cb0cf4649f97556cdc91383d0cf851b350d2035a1de54c83"
Nov 03 22:14:12.511134 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.511118    4431 scope.go:122] "RemoveContainer" containerID="ffd11b079eac9d3158e351174756e300dd22ed9b441f9da11b0c3afded104a04"
Nov 03 22:14:12.517673 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.517655    4431 scope.go:122] "RemoveContainer" containerID="4896d1d8515f07ea7f2021e198adc66c9be14bf35a69397d9bf02e31d8335cd8"
Nov 03 22:14:12.524049 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.524030    4431 scope.go:122] "RemoveContainer" containerID="fa2c478b6f6e2ff47de93363477c6d57a6bd6a5589bfc6ece133d3de475c38ed"
Nov 03 22:14:12.535422 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.535383    4431 scope.go:122] "RemoveContainer" containerID="4bad046ecb8b5b1068e1ec5567a005daa35f9514a191799894a0fc284e418a69"
Nov 03 22:14:12.542558 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.542537    4431 scope.go:122] "RemoveContainer" containerID="6a43f85ad035f95fed9c6dde29c7d6d0442d39bb05a5929de3d411711c512196"
Nov 03 22:14:12.548701 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.548682    4431 scope.go:122] "RemoveContainer" containerID="bf6abfe3e4a0c89c18afb56290886e7a1de70da993232fa0350170875592b1f1"
Nov 03 22:14:12.548990 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:14:12.548962    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf6abfe3e4a0c89c18afb56290886e7a1de70da993232fa0350170875592b1f1\": container with ID starting with bf6abfe3e4a0c89c18afb56290886e7a1de70da993232fa0350170875592b1f1 not found: ID does not exist" containerID="bf6abfe3e4a0c89c18afb56290886e7a1de70da993232fa0350170875592b1f1"
Nov 03 22:14:12.549032 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.548999    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf6abfe3e4a0c89c18afb56290886e7a1de70da993232fa0350170875592b1f1"} err="failed to get container status \"bf6abfe3e4a0c89c18afb56290886e7a1de70da993232fa0350170875592b1f1\": rpc error: code = NotFound desc = could not find container \"bf6abfe3e4a0c89c18afb56290886e7a1de70da993232fa0350170875592b1f1\": container with ID starting with bf6abfe3e4a0c89c18afb56290886e7a1de70da993232fa0350170875592b1f1 not found: ID does not exist"
Nov 03 22:14:12.549075 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.549033    4431 scope.go:122] "RemoveContainer" containerID="b9551a8a95f7a755cb0cf4649f97556cdc91383d0cf851b350d2035a1de54c83"
Nov 03 22:14:12.549308 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:14:12.549291    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9551a8a95f7a755cb0cf4649f97556cdc91383d0cf851b350d2035a1de54c83\": container with ID starting with b9551a8a95f7a755cb0cf4649f97556cdc91383d0cf851b350d2035a1de54c83 not found: ID does not exist" containerID="b9551a8a95f7a755cb0cf4649f97556cdc91383d0cf851b350d2035a1de54c83"
Nov 03 22:14:12.549343 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.549314    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9551a8a95f7a755cb0cf4649f97556cdc91383d0cf851b350d2035a1de54c83"} err="failed to get container status \"b9551a8a95f7a755cb0cf4649f97556cdc91383d0cf851b350d2035a1de54c83\": rpc error: code = NotFound desc = could not find container \"b9551a8a95f7a755cb0cf4649f97556cdc91383d0cf851b350d2035a1de54c83\": container with ID starting with b9551a8a95f7a755cb0cf4649f97556cdc91383d0cf851b350d2035a1de54c83 not found: ID does not exist"
Nov 03 22:14:12.549343 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.549334    4431 scope.go:122] "RemoveContainer" containerID="ffd11b079eac9d3158e351174756e300dd22ed9b441f9da11b0c3afded104a04"
Nov 03 22:14:12.549552 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:14:12.549535    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffd11b079eac9d3158e351174756e300dd22ed9b441f9da11b0c3afded104a04\": container with ID starting with ffd11b079eac9d3158e351174756e300dd22ed9b441f9da11b0c3afded104a04 not found: ID does not exist" containerID="ffd11b079eac9d3158e351174756e300dd22ed9b441f9da11b0c3afded104a04"
Nov 03 22:14:12.549593 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.549557    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffd11b079eac9d3158e351174756e300dd22ed9b441f9da11b0c3afded104a04"} err="failed to get container status \"ffd11b079eac9d3158e351174756e300dd22ed9b441f9da11b0c3afded104a04\": rpc error: code = NotFound desc = could not find container \"ffd11b079eac9d3158e351174756e300dd22ed9b441f9da11b0c3afded104a04\": container with ID starting with ffd11b079eac9d3158e351174756e300dd22ed9b441f9da11b0c3afded104a04 not found: ID does not exist"
Nov 03 22:14:12.549593 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.549571    4431 scope.go:122] "RemoveContainer" containerID="4896d1d8515f07ea7f2021e198adc66c9be14bf35a69397d9bf02e31d8335cd8"
Nov 03 22:14:12.549777 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:14:12.549757    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4896d1d8515f07ea7f2021e198adc66c9be14bf35a69397d9bf02e31d8335cd8\": container with ID starting with 4896d1d8515f07ea7f2021e198adc66c9be14bf35a69397d9bf02e31d8335cd8 not found: ID does not exist" containerID="4896d1d8515f07ea7f2021e198adc66c9be14bf35a69397d9bf02e31d8335cd8"
Nov 03 22:14:12.549810 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.549784    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4896d1d8515f07ea7f2021e198adc66c9be14bf35a69397d9bf02e31d8335cd8"} err="failed to get container status \"4896d1d8515f07ea7f2021e198adc66c9be14bf35a69397d9bf02e31d8335cd8\": rpc error: code = NotFound desc = could not find container \"4896d1d8515f07ea7f2021e198adc66c9be14bf35a69397d9bf02e31d8335cd8\": container with ID starting with 4896d1d8515f07ea7f2021e198adc66c9be14bf35a69397d9bf02e31d8335cd8 not found: ID does not exist"
Nov 03 22:14:12.549810 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.549799    4431 scope.go:122] "RemoveContainer" containerID="fa2c478b6f6e2ff47de93363477c6d57a6bd6a5589bfc6ece133d3de475c38ed"
Nov 03 22:14:12.550017 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:14:12.550003    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa2c478b6f6e2ff47de93363477c6d57a6bd6a5589bfc6ece133d3de475c38ed\": container with ID starting with fa2c478b6f6e2ff47de93363477c6d57a6bd6a5589bfc6ece133d3de475c38ed not found: ID does not exist" containerID="fa2c478b6f6e2ff47de93363477c6d57a6bd6a5589bfc6ece133d3de475c38ed"
Nov 03 22:14:12.550053 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.550021    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa2c478b6f6e2ff47de93363477c6d57a6bd6a5589bfc6ece133d3de475c38ed"} err="failed to get container status \"fa2c478b6f6e2ff47de93363477c6d57a6bd6a5589bfc6ece133d3de475c38ed\": rpc error: code = NotFound desc = could not find container \"fa2c478b6f6e2ff47de93363477c6d57a6bd6a5589bfc6ece133d3de475c38ed\": container with ID starting with fa2c478b6f6e2ff47de93363477c6d57a6bd6a5589bfc6ece133d3de475c38ed not found: ID does not exist"
Nov 03 22:14:12.550053 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.550032    4431 scope.go:122] "RemoveContainer" containerID="4bad046ecb8b5b1068e1ec5567a005daa35f9514a191799894a0fc284e418a69"
Nov 03 22:14:12.550251 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:14:12.550231    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bad046ecb8b5b1068e1ec5567a005daa35f9514a191799894a0fc284e418a69\": container with ID starting with 4bad046ecb8b5b1068e1ec5567a005daa35f9514a191799894a0fc284e418a69 not found: ID does not exist" containerID="4bad046ecb8b5b1068e1ec5567a005daa35f9514a191799894a0fc284e418a69"
Nov 03 22:14:12.550294 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.550260    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bad046ecb8b5b1068e1ec5567a005daa35f9514a191799894a0fc284e418a69"} err="failed to get container status \"4bad046ecb8b5b1068e1ec5567a005daa35f9514a191799894a0fc284e418a69\": rpc error: code = NotFound desc = could not find container \"4bad046ecb8b5b1068e1ec5567a005daa35f9514a191799894a0fc284e418a69\": container with ID starting with 4bad046ecb8b5b1068e1ec5567a005daa35f9514a191799894a0fc284e418a69 not found: ID does not exist"
Nov 03 22:14:12.550294 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.550279    4431 scope.go:122] "RemoveContainer" containerID="6a43f85ad035f95fed9c6dde29c7d6d0442d39bb05a5929de3d411711c512196"
Nov 03 22:14:12.550558 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:14:12.550542    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a43f85ad035f95fed9c6dde29c7d6d0442d39bb05a5929de3d411711c512196\": container with ID starting with 6a43f85ad035f95fed9c6dde29c7d6d0442d39bb05a5929de3d411711c512196 not found: ID does not exist" containerID="6a43f85ad035f95fed9c6dde29c7d6d0442d39bb05a5929de3d411711c512196"
Nov 03 22:14:12.550601 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.550562    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a43f85ad035f95fed9c6dde29c7d6d0442d39bb05a5929de3d411711c512196"} err="failed to get container status \"6a43f85ad035f95fed9c6dde29c7d6d0442d39bb05a5929de3d411711c512196\": rpc error: code = NotFound desc = could not find container \"6a43f85ad035f95fed9c6dde29c7d6d0442d39bb05a5929de3d411711c512196\": container with ID starting with 6a43f85ad035f95fed9c6dde29c7d6d0442d39bb05a5929de3d411711c512196 not found: ID does not exist"
Nov 03 22:14:12.550601 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.550575    4431 scope.go:122] "RemoveContainer" containerID="bf6abfe3e4a0c89c18afb56290886e7a1de70da993232fa0350170875592b1f1"
Nov 03 22:14:12.550785 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.550768    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf6abfe3e4a0c89c18afb56290886e7a1de70da993232fa0350170875592b1f1"} err="failed to get container status \"bf6abfe3e4a0c89c18afb56290886e7a1de70da993232fa0350170875592b1f1\": rpc error: code = NotFound desc = could not find container \"bf6abfe3e4a0c89c18afb56290886e7a1de70da993232fa0350170875592b1f1\": container with ID starting with bf6abfe3e4a0c89c18afb56290886e7a1de70da993232fa0350170875592b1f1 not found: ID does not exist"
Nov 03 22:14:12.550852 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.550785    4431 scope.go:122] "RemoveContainer" containerID="b9551a8a95f7a755cb0cf4649f97556cdc91383d0cf851b350d2035a1de54c83"
Nov 03 22:14:12.550977 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.550962    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9551a8a95f7a755cb0cf4649f97556cdc91383d0cf851b350d2035a1de54c83"} err="failed to get container status \"b9551a8a95f7a755cb0cf4649f97556cdc91383d0cf851b350d2035a1de54c83\": rpc error: code = NotFound desc = could not find container \"b9551a8a95f7a755cb0cf4649f97556cdc91383d0cf851b350d2035a1de54c83\": container with ID starting with b9551a8a95f7a755cb0cf4649f97556cdc91383d0cf851b350d2035a1de54c83 not found: ID does not exist"
Nov 03 22:14:12.551012 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.550978    4431 scope.go:122] "RemoveContainer" containerID="ffd11b079eac9d3158e351174756e300dd22ed9b441f9da11b0c3afded104a04"
Nov 03 22:14:12.551126 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.551113    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffd11b079eac9d3158e351174756e300dd22ed9b441f9da11b0c3afded104a04"} err="failed to get container status \"ffd11b079eac9d3158e351174756e300dd22ed9b441f9da11b0c3afded104a04\": rpc error: code = NotFound desc = could not find container \"ffd11b079eac9d3158e351174756e300dd22ed9b441f9da11b0c3afded104a04\": container with ID starting with ffd11b079eac9d3158e351174756e300dd22ed9b441f9da11b0c3afded104a04 not found: ID does not exist"
Nov 03 22:14:12.551163 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.551126    4431 scope.go:122] "RemoveContainer" containerID="4896d1d8515f07ea7f2021e198adc66c9be14bf35a69397d9bf02e31d8335cd8"
Nov 03 22:14:12.551354 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.551338    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4896d1d8515f07ea7f2021e198adc66c9be14bf35a69397d9bf02e31d8335cd8"} err="failed to get container status \"4896d1d8515f07ea7f2021e198adc66c9be14bf35a69397d9bf02e31d8335cd8\": rpc error: code = NotFound desc = could not find container \"4896d1d8515f07ea7f2021e198adc66c9be14bf35a69397d9bf02e31d8335cd8\": container with ID starting with 4896d1d8515f07ea7f2021e198adc66c9be14bf35a69397d9bf02e31d8335cd8 not found: ID does not exist"
Nov 03 22:14:12.551404 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.551354    4431 scope.go:122] "RemoveContainer" containerID="fa2c478b6f6e2ff47de93363477c6d57a6bd6a5589bfc6ece133d3de475c38ed"
Nov 03 22:14:12.551562 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.551544    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa2c478b6f6e2ff47de93363477c6d57a6bd6a5589bfc6ece133d3de475c38ed"} err="failed to get container status \"fa2c478b6f6e2ff47de93363477c6d57a6bd6a5589bfc6ece133d3de475c38ed\": rpc error: code = NotFound desc = could not find container \"fa2c478b6f6e2ff47de93363477c6d57a6bd6a5589bfc6ece133d3de475c38ed\": container with ID starting with fa2c478b6f6e2ff47de93363477c6d57a6bd6a5589bfc6ece133d3de475c38ed not found: ID does not exist"
Nov 03 22:14:12.551610 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.551562    4431 scope.go:122] "RemoveContainer" containerID="4bad046ecb8b5b1068e1ec5567a005daa35f9514a191799894a0fc284e418a69"
Nov 03 22:14:12.551798 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.551781    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bad046ecb8b5b1068e1ec5567a005daa35f9514a191799894a0fc284e418a69"} err="failed to get container status \"4bad046ecb8b5b1068e1ec5567a005daa35f9514a191799894a0fc284e418a69\": rpc error: code = NotFound desc = could not find container \"4bad046ecb8b5b1068e1ec5567a005daa35f9514a191799894a0fc284e418a69\": container with ID starting with 4bad046ecb8b5b1068e1ec5567a005daa35f9514a191799894a0fc284e418a69 not found: ID does not exist"
Nov 03 22:14:12.551843 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.551798    4431 scope.go:122] "RemoveContainer" containerID="6a43f85ad035f95fed9c6dde29c7d6d0442d39bb05a5929de3d411711c512196"
Nov 03 22:14:12.551991 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.551975    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a43f85ad035f95fed9c6dde29c7d6d0442d39bb05a5929de3d411711c512196"} err="failed to get container status \"6a43f85ad035f95fed9c6dde29c7d6d0442d39bb05a5929de3d411711c512196\": rpc error: code = NotFound desc = could not find container \"6a43f85ad035f95fed9c6dde29c7d6d0442d39bb05a5929de3d411711c512196\": container with ID starting with 6a43f85ad035f95fed9c6dde29c7d6d0442d39bb05a5929de3d411711c512196 not found: ID does not exist"
Nov 03 22:14:12.552036 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.551990    4431 scope.go:122] "RemoveContainer" containerID="bf6abfe3e4a0c89c18afb56290886e7a1de70da993232fa0350170875592b1f1"
Nov 03 22:14:12.552187 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.552171    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf6abfe3e4a0c89c18afb56290886e7a1de70da993232fa0350170875592b1f1"} err="failed to get container status \"bf6abfe3e4a0c89c18afb56290886e7a1de70da993232fa0350170875592b1f1\": rpc error: code = NotFound desc = could not find container \"bf6abfe3e4a0c89c18afb56290886e7a1de70da993232fa0350170875592b1f1\": container with ID starting with bf6abfe3e4a0c89c18afb56290886e7a1de70da993232fa0350170875592b1f1 not found: ID does not exist"
Nov 03 22:14:12.552242 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.552187    4431 scope.go:122] "RemoveContainer" containerID="b9551a8a95f7a755cb0cf4649f97556cdc91383d0cf851b350d2035a1de54c83"
Nov 03 22:14:12.552377 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.552361    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9551a8a95f7a755cb0cf4649f97556cdc91383d0cf851b350d2035a1de54c83"} err="failed to get container status \"b9551a8a95f7a755cb0cf4649f97556cdc91383d0cf851b350d2035a1de54c83\": rpc error: code = NotFound desc = could not find container \"b9551a8a95f7a755cb0cf4649f97556cdc91383d0cf851b350d2035a1de54c83\": container with ID starting with b9551a8a95f7a755cb0cf4649f97556cdc91383d0cf851b350d2035a1de54c83 not found: ID does not exist"
Nov 03 22:14:12.552419 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.552377    4431 scope.go:122] "RemoveContainer" containerID="ffd11b079eac9d3158e351174756e300dd22ed9b441f9da11b0c3afded104a04"
Nov 03 22:14:12.552558 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.552544    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffd11b079eac9d3158e351174756e300dd22ed9b441f9da11b0c3afded104a04"} err="failed to get container status \"ffd11b079eac9d3158e351174756e300dd22ed9b441f9da11b0c3afded104a04\": rpc error: code = NotFound desc = could not find container \"ffd11b079eac9d3158e351174756e300dd22ed9b441f9da11b0c3afded104a04\": container with ID starting with ffd11b079eac9d3158e351174756e300dd22ed9b441f9da11b0c3afded104a04 not found: ID does not exist"
Nov 03 22:14:12.552593 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.552558    4431 scope.go:122] "RemoveContainer" containerID="4896d1d8515f07ea7f2021e198adc66c9be14bf35a69397d9bf02e31d8335cd8"
Nov 03 22:14:12.552761 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.552746    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4896d1d8515f07ea7f2021e198adc66c9be14bf35a69397d9bf02e31d8335cd8"} err="failed to get container status \"4896d1d8515f07ea7f2021e198adc66c9be14bf35a69397d9bf02e31d8335cd8\": rpc error: code = NotFound desc = could not find container \"4896d1d8515f07ea7f2021e198adc66c9be14bf35a69397d9bf02e31d8335cd8\": container with ID starting with 4896d1d8515f07ea7f2021e198adc66c9be14bf35a69397d9bf02e31d8335cd8 not found: ID does not exist"
Nov 03 22:14:12.552807 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.552761    4431 scope.go:122] "RemoveContainer" containerID="fa2c478b6f6e2ff47de93363477c6d57a6bd6a5589bfc6ece133d3de475c38ed"
Nov 03 22:14:12.552956 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.552941    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa2c478b6f6e2ff47de93363477c6d57a6bd6a5589bfc6ece133d3de475c38ed"} err="failed to get container status \"fa2c478b6f6e2ff47de93363477c6d57a6bd6a5589bfc6ece133d3de475c38ed\": rpc error: code = NotFound desc = could not find container \"fa2c478b6f6e2ff47de93363477c6d57a6bd6a5589bfc6ece133d3de475c38ed\": container with ID starting with fa2c478b6f6e2ff47de93363477c6d57a6bd6a5589bfc6ece133d3de475c38ed not found: ID does not exist"
Nov 03 22:14:12.552997 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.552956    4431 scope.go:122] "RemoveContainer" containerID="4bad046ecb8b5b1068e1ec5567a005daa35f9514a191799894a0fc284e418a69"
Nov 03 22:14:12.553147 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.553131    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bad046ecb8b5b1068e1ec5567a005daa35f9514a191799894a0fc284e418a69"} err="failed to get container status \"4bad046ecb8b5b1068e1ec5567a005daa35f9514a191799894a0fc284e418a69\": rpc error: code = NotFound desc = could not find container \"4bad046ecb8b5b1068e1ec5567a005daa35f9514a191799894a0fc284e418a69\": container with ID starting with 4bad046ecb8b5b1068e1ec5567a005daa35f9514a191799894a0fc284e418a69 not found: ID does not exist"
Nov 03 22:14:12.553192 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.553147    4431 scope.go:122] "RemoveContainer" containerID="6a43f85ad035f95fed9c6dde29c7d6d0442d39bb05a5929de3d411711c512196"
Nov 03 22:14:12.553350 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.553328    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a43f85ad035f95fed9c6dde29c7d6d0442d39bb05a5929de3d411711c512196"} err="failed to get container status \"6a43f85ad035f95fed9c6dde29c7d6d0442d39bb05a5929de3d411711c512196\": rpc error: code = NotFound desc = could not find container \"6a43f85ad035f95fed9c6dde29c7d6d0442d39bb05a5929de3d411711c512196\": container with ID starting with 6a43f85ad035f95fed9c6dde29c7d6d0442d39bb05a5929de3d411711c512196 not found: ID does not exist"
Nov 03 22:14:12.553417 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.553350    4431 scope.go:122] "RemoveContainer" containerID="bf6abfe3e4a0c89c18afb56290886e7a1de70da993232fa0350170875592b1f1"
Nov 03 22:14:12.553578 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.553562    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf6abfe3e4a0c89c18afb56290886e7a1de70da993232fa0350170875592b1f1"} err="failed to get container status \"bf6abfe3e4a0c89c18afb56290886e7a1de70da993232fa0350170875592b1f1\": rpc error: code = NotFound desc = could not find container \"bf6abfe3e4a0c89c18afb56290886e7a1de70da993232fa0350170875592b1f1\": container with ID starting with bf6abfe3e4a0c89c18afb56290886e7a1de70da993232fa0350170875592b1f1 not found: ID does not exist"
Nov 03 22:14:12.553627 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.553578    4431 scope.go:122] "RemoveContainer" containerID="b9551a8a95f7a755cb0cf4649f97556cdc91383d0cf851b350d2035a1de54c83"
Nov 03 22:14:12.553761 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.553745    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9551a8a95f7a755cb0cf4649f97556cdc91383d0cf851b350d2035a1de54c83"} err="failed to get container status \"b9551a8a95f7a755cb0cf4649f97556cdc91383d0cf851b350d2035a1de54c83\": rpc error: code = NotFound desc = could not find container \"b9551a8a95f7a755cb0cf4649f97556cdc91383d0cf851b350d2035a1de54c83\": container with ID starting with b9551a8a95f7a755cb0cf4649f97556cdc91383d0cf851b350d2035a1de54c83 not found: ID does not exist"
Nov 03 22:14:12.553801 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.553761    4431 scope.go:122] "RemoveContainer" containerID="ffd11b079eac9d3158e351174756e300dd22ed9b441f9da11b0c3afded104a04"
Nov 03 22:14:12.553954 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.553938    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffd11b079eac9d3158e351174756e300dd22ed9b441f9da11b0c3afded104a04"} err="failed to get container status \"ffd11b079eac9d3158e351174756e300dd22ed9b441f9da11b0c3afded104a04\": rpc error: code = NotFound desc = could not find container \"ffd11b079eac9d3158e351174756e300dd22ed9b441f9da11b0c3afded104a04\": container with ID starting with ffd11b079eac9d3158e351174756e300dd22ed9b441f9da11b0c3afded104a04 not found: ID does not exist"
Nov 03 22:14:12.553990 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.553953    4431 scope.go:122] "RemoveContainer" containerID="4896d1d8515f07ea7f2021e198adc66c9be14bf35a69397d9bf02e31d8335cd8"
Nov 03 22:14:12.554139 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.554123    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4896d1d8515f07ea7f2021e198adc66c9be14bf35a69397d9bf02e31d8335cd8"} err="failed to get container status \"4896d1d8515f07ea7f2021e198adc66c9be14bf35a69397d9bf02e31d8335cd8\": rpc error: code = NotFound desc = could not find container \"4896d1d8515f07ea7f2021e198adc66c9be14bf35a69397d9bf02e31d8335cd8\": container with ID starting with 4896d1d8515f07ea7f2021e198adc66c9be14bf35a69397d9bf02e31d8335cd8 not found: ID does not exist"
Nov 03 22:14:12.554200 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.554140    4431 scope.go:122] "RemoveContainer" containerID="fa2c478b6f6e2ff47de93363477c6d57a6bd6a5589bfc6ece133d3de475c38ed"
Nov 03 22:14:12.554450 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.554430    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa2c478b6f6e2ff47de93363477c6d57a6bd6a5589bfc6ece133d3de475c38ed"} err="failed to get container status \"fa2c478b6f6e2ff47de93363477c6d57a6bd6a5589bfc6ece133d3de475c38ed\": rpc error: code = NotFound desc = could not find container \"fa2c478b6f6e2ff47de93363477c6d57a6bd6a5589bfc6ece133d3de475c38ed\": container with ID starting with fa2c478b6f6e2ff47de93363477c6d57a6bd6a5589bfc6ece133d3de475c38ed not found: ID does not exist"
Nov 03 22:14:12.554497 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.554450    4431 scope.go:122] "RemoveContainer" containerID="4bad046ecb8b5b1068e1ec5567a005daa35f9514a191799894a0fc284e418a69"
Nov 03 22:14:12.554650 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.554631    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bad046ecb8b5b1068e1ec5567a005daa35f9514a191799894a0fc284e418a69"} err="failed to get container status \"4bad046ecb8b5b1068e1ec5567a005daa35f9514a191799894a0fc284e418a69\": rpc error: code = NotFound desc = could not find container \"4bad046ecb8b5b1068e1ec5567a005daa35f9514a191799894a0fc284e418a69\": container with ID starting with 4bad046ecb8b5b1068e1ec5567a005daa35f9514a191799894a0fc284e418a69 not found: ID does not exist"
Nov 03 22:14:12.554710 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.554650    4431 scope.go:122] "RemoveContainer" containerID="6a43f85ad035f95fed9c6dde29c7d6d0442d39bb05a5929de3d411711c512196"
Nov 03 22:14:12.554849 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.554829    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a43f85ad035f95fed9c6dde29c7d6d0442d39bb05a5929de3d411711c512196"} err="failed to get container status \"6a43f85ad035f95fed9c6dde29c7d6d0442d39bb05a5929de3d411711c512196\": rpc error: code = NotFound desc = could not find container \"6a43f85ad035f95fed9c6dde29c7d6d0442d39bb05a5929de3d411711c512196\": container with ID starting with 6a43f85ad035f95fed9c6dde29c7d6d0442d39bb05a5929de3d411711c512196 not found: ID does not exist"
Nov 03 22:14:12.554890 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.554850    4431 scope.go:122] "RemoveContainer" containerID="bf6abfe3e4a0c89c18afb56290886e7a1de70da993232fa0350170875592b1f1"
Nov 03 22:14:12.555019 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.555002    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf6abfe3e4a0c89c18afb56290886e7a1de70da993232fa0350170875592b1f1"} err="failed to get container status \"bf6abfe3e4a0c89c18afb56290886e7a1de70da993232fa0350170875592b1f1\": rpc error: code = NotFound desc = could not find container \"bf6abfe3e4a0c89c18afb56290886e7a1de70da993232fa0350170875592b1f1\": container with ID starting with bf6abfe3e4a0c89c18afb56290886e7a1de70da993232fa0350170875592b1f1 not found: ID does not exist"
Nov 03 22:14:12.555057 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.555020    4431 scope.go:122] "RemoveContainer" containerID="b9551a8a95f7a755cb0cf4649f97556cdc91383d0cf851b350d2035a1de54c83"
Nov 03 22:14:12.555259 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.555242    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9551a8a95f7a755cb0cf4649f97556cdc91383d0cf851b350d2035a1de54c83"} err="failed to get container status \"b9551a8a95f7a755cb0cf4649f97556cdc91383d0cf851b350d2035a1de54c83\": rpc error: code = NotFound desc = could not find container \"b9551a8a95f7a755cb0cf4649f97556cdc91383d0cf851b350d2035a1de54c83\": container with ID starting with b9551a8a95f7a755cb0cf4649f97556cdc91383d0cf851b350d2035a1de54c83 not found: ID does not exist"
Nov 03 22:14:12.555349 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.555259    4431 scope.go:122] "RemoveContainer" containerID="ffd11b079eac9d3158e351174756e300dd22ed9b441f9da11b0c3afded104a04"
Nov 03 22:14:12.555467 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.555449    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffd11b079eac9d3158e351174756e300dd22ed9b441f9da11b0c3afded104a04"} err="failed to get container status \"ffd11b079eac9d3158e351174756e300dd22ed9b441f9da11b0c3afded104a04\": rpc error: code = NotFound desc = could not find container \"ffd11b079eac9d3158e351174756e300dd22ed9b441f9da11b0c3afded104a04\": container with ID starting with ffd11b079eac9d3158e351174756e300dd22ed9b441f9da11b0c3afded104a04 not found: ID does not exist"
Nov 03 22:14:12.555511 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.555492    4431 scope.go:122] "RemoveContainer" containerID="4896d1d8515f07ea7f2021e198adc66c9be14bf35a69397d9bf02e31d8335cd8"
Nov 03 22:14:12.555739 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.555722    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4896d1d8515f07ea7f2021e198adc66c9be14bf35a69397d9bf02e31d8335cd8"} err="failed to get container status \"4896d1d8515f07ea7f2021e198adc66c9be14bf35a69397d9bf02e31d8335cd8\": rpc error: code = NotFound desc = could not find container \"4896d1d8515f07ea7f2021e198adc66c9be14bf35a69397d9bf02e31d8335cd8\": container with ID starting with 4896d1d8515f07ea7f2021e198adc66c9be14bf35a69397d9bf02e31d8335cd8 not found: ID does not exist"
Nov 03 22:14:12.555739 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.555738    4431 scope.go:122] "RemoveContainer" containerID="fa2c478b6f6e2ff47de93363477c6d57a6bd6a5589bfc6ece133d3de475c38ed"
Nov 03 22:14:12.556000 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.555983    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa2c478b6f6e2ff47de93363477c6d57a6bd6a5589bfc6ece133d3de475c38ed"} err="failed to get container status \"fa2c478b6f6e2ff47de93363477c6d57a6bd6a5589bfc6ece133d3de475c38ed\": rpc error: code = NotFound desc = could not find container \"fa2c478b6f6e2ff47de93363477c6d57a6bd6a5589bfc6ece133d3de475c38ed\": container with ID starting with fa2c478b6f6e2ff47de93363477c6d57a6bd6a5589bfc6ece133d3de475c38ed not found: ID does not exist"
Nov 03 22:14:12.556000 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.555999    4431 scope.go:122] "RemoveContainer" containerID="4bad046ecb8b5b1068e1ec5567a005daa35f9514a191799894a0fc284e418a69"
Nov 03 22:14:12.556239 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.556205    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bad046ecb8b5b1068e1ec5567a005daa35f9514a191799894a0fc284e418a69"} err="failed to get container status \"4bad046ecb8b5b1068e1ec5567a005daa35f9514a191799894a0fc284e418a69\": rpc error: code = NotFound desc = could not find container \"4bad046ecb8b5b1068e1ec5567a005daa35f9514a191799894a0fc284e418a69\": container with ID starting with 4bad046ecb8b5b1068e1ec5567a005daa35f9514a191799894a0fc284e418a69 not found: ID does not exist"
Nov 03 22:14:12.556288 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.556240    4431 scope.go:122] "RemoveContainer" containerID="6a43f85ad035f95fed9c6dde29c7d6d0442d39bb05a5929de3d411711c512196"
Nov 03 22:14:12.556471 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.556454    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a43f85ad035f95fed9c6dde29c7d6d0442d39bb05a5929de3d411711c512196"} err="failed to get container status \"6a43f85ad035f95fed9c6dde29c7d6d0442d39bb05a5929de3d411711c512196\": rpc error: code = NotFound desc = could not find container \"6a43f85ad035f95fed9c6dde29c7d6d0442d39bb05a5929de3d411711c512196\": container with ID starting with 6a43f85ad035f95fed9c6dde29c7d6d0442d39bb05a5929de3d411711c512196 not found: ID does not exist"
Nov 03 22:14:12.556471 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.556471    4431 scope.go:122] "RemoveContainer" containerID="bf6abfe3e4a0c89c18afb56290886e7a1de70da993232fa0350170875592b1f1"
Nov 03 22:14:12.556686 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.556668    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf6abfe3e4a0c89c18afb56290886e7a1de70da993232fa0350170875592b1f1"} err="failed to get container status \"bf6abfe3e4a0c89c18afb56290886e7a1de70da993232fa0350170875592b1f1\": rpc error: code = NotFound desc = could not find container \"bf6abfe3e4a0c89c18afb56290886e7a1de70da993232fa0350170875592b1f1\": container with ID starting with bf6abfe3e4a0c89c18afb56290886e7a1de70da993232fa0350170875592b1f1 not found: ID does not exist"
Nov 03 22:14:12.556744 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.556687    4431 scope.go:122] "RemoveContainer" containerID="b9551a8a95f7a755cb0cf4649f97556cdc91383d0cf851b350d2035a1de54c83"
Nov 03 22:14:12.556894 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.556873    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9551a8a95f7a755cb0cf4649f97556cdc91383d0cf851b350d2035a1de54c83"} err="failed to get container status \"b9551a8a95f7a755cb0cf4649f97556cdc91383d0cf851b350d2035a1de54c83\": rpc error: code = NotFound desc = could not find container \"b9551a8a95f7a755cb0cf4649f97556cdc91383d0cf851b350d2035a1de54c83\": container with ID starting with b9551a8a95f7a755cb0cf4649f97556cdc91383d0cf851b350d2035a1de54c83 not found: ID does not exist"
Nov 03 22:14:12.556946 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.556894    4431 scope.go:122] "RemoveContainer" containerID="ffd11b079eac9d3158e351174756e300dd22ed9b441f9da11b0c3afded104a04"
Nov 03 22:14:12.557096 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.557079    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffd11b079eac9d3158e351174756e300dd22ed9b441f9da11b0c3afded104a04"} err="failed to get container status \"ffd11b079eac9d3158e351174756e300dd22ed9b441f9da11b0c3afded104a04\": rpc error: code = NotFound desc = could not find container \"ffd11b079eac9d3158e351174756e300dd22ed9b441f9da11b0c3afded104a04\": container with ID starting with ffd11b079eac9d3158e351174756e300dd22ed9b441f9da11b0c3afded104a04 not found: ID does not exist"
Nov 03 22:14:12.557138 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.557096    4431 scope.go:122] "RemoveContainer" containerID="4896d1d8515f07ea7f2021e198adc66c9be14bf35a69397d9bf02e31d8335cd8"
Nov 03 22:14:12.557344 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.557325    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4896d1d8515f07ea7f2021e198adc66c9be14bf35a69397d9bf02e31d8335cd8"} err="failed to get container status \"4896d1d8515f07ea7f2021e198adc66c9be14bf35a69397d9bf02e31d8335cd8\": rpc error: code = NotFound desc = could not find container \"4896d1d8515f07ea7f2021e198adc66c9be14bf35a69397d9bf02e31d8335cd8\": container with ID starting with 4896d1d8515f07ea7f2021e198adc66c9be14bf35a69397d9bf02e31d8335cd8 not found: ID does not exist"
Nov 03 22:14:12.557392 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.557345    4431 scope.go:122] "RemoveContainer" containerID="fa2c478b6f6e2ff47de93363477c6d57a6bd6a5589bfc6ece133d3de475c38ed"
Nov 03 22:14:12.557545 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.557526    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa2c478b6f6e2ff47de93363477c6d57a6bd6a5589bfc6ece133d3de475c38ed"} err="failed to get container status \"fa2c478b6f6e2ff47de93363477c6d57a6bd6a5589bfc6ece133d3de475c38ed\": rpc error: code = NotFound desc = could not find container \"fa2c478b6f6e2ff47de93363477c6d57a6bd6a5589bfc6ece133d3de475c38ed\": container with ID starting with fa2c478b6f6e2ff47de93363477c6d57a6bd6a5589bfc6ece133d3de475c38ed not found: ID does not exist"
Nov 03 22:14:12.557581 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.557545    4431 scope.go:122] "RemoveContainer" containerID="4bad046ecb8b5b1068e1ec5567a005daa35f9514a191799894a0fc284e418a69"
Nov 03 22:14:12.557742 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.557725    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bad046ecb8b5b1068e1ec5567a005daa35f9514a191799894a0fc284e418a69"} err="failed to get container status \"4bad046ecb8b5b1068e1ec5567a005daa35f9514a191799894a0fc284e418a69\": rpc error: code = NotFound desc = could not find container \"4bad046ecb8b5b1068e1ec5567a005daa35f9514a191799894a0fc284e418a69\": container with ID starting with 4bad046ecb8b5b1068e1ec5567a005daa35f9514a191799894a0fc284e418a69 not found: ID does not exist"
Nov 03 22:14:12.557779 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.557742    4431 scope.go:122] "RemoveContainer" containerID="6a43f85ad035f95fed9c6dde29c7d6d0442d39bb05a5929de3d411711c512196"
Nov 03 22:14:12.557964 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.557946    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a43f85ad035f95fed9c6dde29c7d6d0442d39bb05a5929de3d411711c512196"} err="failed to get container status \"6a43f85ad035f95fed9c6dde29c7d6d0442d39bb05a5929de3d411711c512196\": rpc error: code = NotFound desc = could not find container \"6a43f85ad035f95fed9c6dde29c7d6d0442d39bb05a5929de3d411711c512196\": container with ID starting with 6a43f85ad035f95fed9c6dde29c7d6d0442d39bb05a5929de3d411711c512196 not found: ID does not exist"
Nov 03 22:14:12.573032 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.572979    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.560337449 podStartE2EDuration="3.572965491s" totalImagesPullingTime="0s" totalInitContainerRuntime="12.628042ms" isStatefulPod=false podCreationTimestamp="2026-11-03 22:14:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:14:12.572934988 +0000 UTC m=+76.598790851" watchObservedRunningTime="2026-11-03 22:14:12.572965491 +0000 UTC m=+76.598821353"
Nov 03 22:14:12.611112 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.611084    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 22:14:12.614541 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.614513    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 22:14:12.646790 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.646759    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 22:14:12.647070 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.647057    4431 cpu_manager.go:414] "containerMap: removing container" podUID="1e33b576-1208-4325-87e9-4351920bfa06" containerName="init-config-reloader"
Nov 03 22:14:12.647127 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.647073    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e33b576-1208-4325-87e9-4351920bfa06" containerName="init-config-reloader"
Nov 03 22:14:12.647127 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.647086    4431 cpu_manager.go:414] "containerMap: removing container" podUID="1e33b576-1208-4325-87e9-4351920bfa06" containerName="config-reloader"
Nov 03 22:14:12.647127 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.647092    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e33b576-1208-4325-87e9-4351920bfa06" containerName="config-reloader"
Nov 03 22:14:12.647127 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.647099    4431 cpu_manager.go:414] "containerMap: removing container" podUID="1e33b576-1208-4325-87e9-4351920bfa06" containerName="thanos-sidecar"
Nov 03 22:14:12.647127 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.647104    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e33b576-1208-4325-87e9-4351920bfa06" containerName="thanos-sidecar"
Nov 03 22:14:12.647127 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.647119    4431 cpu_manager.go:414] "containerMap: removing container" podUID="1e33b576-1208-4325-87e9-4351920bfa06" containerName="kube-rbac-proxy"
Nov 03 22:14:12.647127 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.647124    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e33b576-1208-4325-87e9-4351920bfa06" containerName="kube-rbac-proxy"
Nov 03 22:14:12.647350 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.647133    4431 cpu_manager.go:414] "containerMap: removing container" podUID="1e33b576-1208-4325-87e9-4351920bfa06" containerName="kube-rbac-proxy-web"
Nov 03 22:14:12.647350 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.647138    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e33b576-1208-4325-87e9-4351920bfa06" containerName="kube-rbac-proxy-web"
Nov 03 22:14:12.647350 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.647148    4431 cpu_manager.go:414] "containerMap: removing container" podUID="1e33b576-1208-4325-87e9-4351920bfa06" containerName="kube-rbac-proxy-thanos"
Nov 03 22:14:12.647350 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.647155    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e33b576-1208-4325-87e9-4351920bfa06" containerName="kube-rbac-proxy-thanos"
Nov 03 22:14:12.647350 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.647164    4431 cpu_manager.go:414] "containerMap: removing container" podUID="1e33b576-1208-4325-87e9-4351920bfa06" containerName="prometheus"
Nov 03 22:14:12.647350 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.647169    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e33b576-1208-4325-87e9-4351920bfa06" containerName="prometheus"
Nov 03 22:14:12.647350 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.647238    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e33b576-1208-4325-87e9-4351920bfa06" containerName="kube-rbac-proxy-thanos"
Nov 03 22:14:12.647350 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.647247    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e33b576-1208-4325-87e9-4351920bfa06" containerName="prometheus"
Nov 03 22:14:12.647350 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.647281    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e33b576-1208-4325-87e9-4351920bfa06" containerName="config-reloader"
Nov 03 22:14:12.647350 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.647287    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e33b576-1208-4325-87e9-4351920bfa06" containerName="thanos-sidecar"
Nov 03 22:14:12.647350 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.647295    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e33b576-1208-4325-87e9-4351920bfa06" containerName="kube-rbac-proxy"
Nov 03 22:14:12.647350 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.647305    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e33b576-1208-4325-87e9-4351920bfa06" containerName="kube-rbac-proxy-web"
Nov 03 22:14:12.669296 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.669273    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:14:12.679604 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.679580    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 22:14:12.684748 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.684721    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Nov 03 22:14:12.684748 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.684723    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Nov 03 22:14:12.685084 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.684839    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 03 22:14:12.685084 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.684845    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Nov 03 22:14:12.685084 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.684977    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Nov 03 22:14:12.685084 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.685006    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Nov 03 22:14:12.685084 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.685025    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Nov 03 22:14:12.685084 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.685029    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Nov 03 22:14:12.685084 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.685041    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Nov 03 22:14:12.685084 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.685057    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Nov 03 22:14:12.685406 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.685104    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Nov 03 22:14:12.685406 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.685161    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-eviee284905oq\""
Nov 03 22:14:12.685496 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.685475    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Nov 03 22:14:12.685538 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.685513    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-2hcp9\""
Nov 03 22:14:12.685573 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.685539    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Nov 03 22:14:12.698040 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.698007    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Nov 03 22:14:12.698040 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.698024    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Nov 03 22:14:12.761476 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.761400    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6e0203c3-558e-4e70-93ec-cd84edfe35e1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6e0203c3-558e-4e70-93ec-cd84edfe35e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:14:12.761476 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.761435    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6e0203c3-558e-4e70-93ec-cd84edfe35e1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6e0203c3-558e-4e70-93ec-cd84edfe35e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:14:12.761476 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.761457    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e0203c3-558e-4e70-93ec-cd84edfe35e1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6e0203c3-558e-4e70-93ec-cd84edfe35e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:14:12.761476 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.761477    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz6zk\" (UniqueName: \"kubernetes.io/projected/6e0203c3-558e-4e70-93ec-cd84edfe35e1-kube-api-access-fz6zk\") pod \"prometheus-k8s-0\" (UID: \"6e0203c3-558e-4e70-93ec-cd84edfe35e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:14:12.761782 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.761576    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6e0203c3-558e-4e70-93ec-cd84edfe35e1-web-config\") pod \"prometheus-k8s-0\" (UID: \"6e0203c3-558e-4e70-93ec-cd84edfe35e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:14:12.761782 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.761672    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e0203c3-558e-4e70-93ec-cd84edfe35e1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6e0203c3-558e-4e70-93ec-cd84edfe35e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:14:12.761782 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.761696    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6e0203c3-558e-4e70-93ec-cd84edfe35e1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6e0203c3-558e-4e70-93ec-cd84edfe35e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:14:12.761782 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.761715    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6e0203c3-558e-4e70-93ec-cd84edfe35e1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6e0203c3-558e-4e70-93ec-cd84edfe35e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:14:12.761782 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.761740    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e0203c3-558e-4e70-93ec-cd84edfe35e1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6e0203c3-558e-4e70-93ec-cd84edfe35e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:14:12.761995 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.761804    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6e0203c3-558e-4e70-93ec-cd84edfe35e1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6e0203c3-558e-4e70-93ec-cd84edfe35e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:14:12.761995 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.761841    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6e0203c3-558e-4e70-93ec-cd84edfe35e1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6e0203c3-558e-4e70-93ec-cd84edfe35e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:14:12.761995 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.761892    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6e0203c3-558e-4e70-93ec-cd84edfe35e1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6e0203c3-558e-4e70-93ec-cd84edfe35e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:14:12.761995 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.761912    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e0203c3-558e-4e70-93ec-cd84edfe35e1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6e0203c3-558e-4e70-93ec-cd84edfe35e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:14:12.761995 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.761927    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6e0203c3-558e-4e70-93ec-cd84edfe35e1-config-out\") pod \"prometheus-k8s-0\" (UID: \"6e0203c3-558e-4e70-93ec-cd84edfe35e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:14:12.761995 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.761958    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6e0203c3-558e-4e70-93ec-cd84edfe35e1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6e0203c3-558e-4e70-93ec-cd84edfe35e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:14:12.761995 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.761994    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6e0203c3-558e-4e70-93ec-cd84edfe35e1-config\") pod \"prometheus-k8s-0\" (UID: \"6e0203c3-558e-4e70-93ec-cd84edfe35e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:14:12.762271 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.762040    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6e0203c3-558e-4e70-93ec-cd84edfe35e1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6e0203c3-558e-4e70-93ec-cd84edfe35e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:14:12.762271 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.762085    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6e0203c3-558e-4e70-93ec-cd84edfe35e1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6e0203c3-558e-4e70-93ec-cd84edfe35e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:14:12.762271 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.762116    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6e0203c3-558e-4e70-93ec-cd84edfe35e1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6e0203c3-558e-4e70-93ec-cd84edfe35e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:14:12.762271 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.762135    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6e0203c3-558e-4e70-93ec-cd84edfe35e1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6e0203c3-558e-4e70-93ec-cd84edfe35e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:14:12.863403 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.863369    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz6zk\" (UniqueName: \"kubernetes.io/projected/6e0203c3-558e-4e70-93ec-cd84edfe35e1-kube-api-access-fz6zk\") pod \"prometheus-k8s-0\" (UID: \"6e0203c3-558e-4e70-93ec-cd84edfe35e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:14:12.863587 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.863402    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6e0203c3-558e-4e70-93ec-cd84edfe35e1-web-config\") pod \"prometheus-k8s-0\" (UID: \"6e0203c3-558e-4e70-93ec-cd84edfe35e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:14:12.863587 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.863461    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e0203c3-558e-4e70-93ec-cd84edfe35e1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6e0203c3-558e-4e70-93ec-cd84edfe35e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:14:12.863587 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.863479    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6e0203c3-558e-4e70-93ec-cd84edfe35e1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6e0203c3-558e-4e70-93ec-cd84edfe35e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:14:12.863587 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.863496    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6e0203c3-558e-4e70-93ec-cd84edfe35e1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6e0203c3-558e-4e70-93ec-cd84edfe35e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:14:12.863587 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.863510    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e0203c3-558e-4e70-93ec-cd84edfe35e1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6e0203c3-558e-4e70-93ec-cd84edfe35e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:14:12.863587 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.863534    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6e0203c3-558e-4e70-93ec-cd84edfe35e1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6e0203c3-558e-4e70-93ec-cd84edfe35e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:14:12.863889 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.863593    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6e0203c3-558e-4e70-93ec-cd84edfe35e1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6e0203c3-558e-4e70-93ec-cd84edfe35e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:14:12.863889 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.863657    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6e0203c3-558e-4e70-93ec-cd84edfe35e1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6e0203c3-558e-4e70-93ec-cd84edfe35e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:14:12.863889 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.863690    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e0203c3-558e-4e70-93ec-cd84edfe35e1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6e0203c3-558e-4e70-93ec-cd84edfe35e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:14:12.863889 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.863712    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6e0203c3-558e-4e70-93ec-cd84edfe35e1-config-out\") pod \"prometheus-k8s-0\" (UID: \"6e0203c3-558e-4e70-93ec-cd84edfe35e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:14:12.863889 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.863734    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6e0203c3-558e-4e70-93ec-cd84edfe35e1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6e0203c3-558e-4e70-93ec-cd84edfe35e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:14:12.863889 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.863759    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6e0203c3-558e-4e70-93ec-cd84edfe35e1-config\") pod \"prometheus-k8s-0\" (UID: \"6e0203c3-558e-4e70-93ec-cd84edfe35e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:14:12.863889 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.863787    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6e0203c3-558e-4e70-93ec-cd84edfe35e1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6e0203c3-558e-4e70-93ec-cd84edfe35e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:14:12.863889 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.863819    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6e0203c3-558e-4e70-93ec-cd84edfe35e1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6e0203c3-558e-4e70-93ec-cd84edfe35e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:14:12.863889 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.863852    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6e0203c3-558e-4e70-93ec-cd84edfe35e1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6e0203c3-558e-4e70-93ec-cd84edfe35e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:14:12.863889 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.863882    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6e0203c3-558e-4e70-93ec-cd84edfe35e1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6e0203c3-558e-4e70-93ec-cd84edfe35e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:14:12.864408 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.863931    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6e0203c3-558e-4e70-93ec-cd84edfe35e1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6e0203c3-558e-4e70-93ec-cd84edfe35e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:14:12.864408 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.863972    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6e0203c3-558e-4e70-93ec-cd84edfe35e1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6e0203c3-558e-4e70-93ec-cd84edfe35e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:14:12.864408 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.864001    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e0203c3-558e-4e70-93ec-cd84edfe35e1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6e0203c3-558e-4e70-93ec-cd84edfe35e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:14:12.864408 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.864035    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6e0203c3-558e-4e70-93ec-cd84edfe35e1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6e0203c3-558e-4e70-93ec-cd84edfe35e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:14:12.864408 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.864386    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e0203c3-558e-4e70-93ec-cd84edfe35e1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6e0203c3-558e-4e70-93ec-cd84edfe35e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:14:12.864680 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.864532    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e0203c3-558e-4e70-93ec-cd84edfe35e1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6e0203c3-558e-4e70-93ec-cd84edfe35e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:14:12.864752 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.864735    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e0203c3-558e-4e70-93ec-cd84edfe35e1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6e0203c3-558e-4e70-93ec-cd84edfe35e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:14:12.864809 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.864792    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e0203c3-558e-4e70-93ec-cd84edfe35e1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6e0203c3-558e-4e70-93ec-cd84edfe35e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:14:12.866915 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.866887    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6e0203c3-558e-4e70-93ec-cd84edfe35e1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6e0203c3-558e-4e70-93ec-cd84edfe35e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:14:12.867038 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.867018    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6e0203c3-558e-4e70-93ec-cd84edfe35e1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6e0203c3-558e-4e70-93ec-cd84edfe35e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:14:12.867417 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.867391    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6e0203c3-558e-4e70-93ec-cd84edfe35e1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6e0203c3-558e-4e70-93ec-cd84edfe35e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:14:12.867575 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.867548    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6e0203c3-558e-4e70-93ec-cd84edfe35e1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6e0203c3-558e-4e70-93ec-cd84edfe35e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:14:12.867647 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.867579    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6e0203c3-558e-4e70-93ec-cd84edfe35e1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6e0203c3-558e-4e70-93ec-cd84edfe35e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:14:12.867647 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.867618    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6e0203c3-558e-4e70-93ec-cd84edfe35e1-web-config\") pod \"prometheus-k8s-0\" (UID: \"6e0203c3-558e-4e70-93ec-cd84edfe35e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:14:12.867834 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.867811    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6e0203c3-558e-4e70-93ec-cd84edfe35e1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6e0203c3-558e-4e70-93ec-cd84edfe35e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:14:12.868047 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.868022    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6e0203c3-558e-4e70-93ec-cd84edfe35e1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6e0203c3-558e-4e70-93ec-cd84edfe35e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:14:12.868309 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.868285    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6e0203c3-558e-4e70-93ec-cd84edfe35e1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6e0203c3-558e-4e70-93ec-cd84edfe35e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:14:12.869150 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.869126    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6e0203c3-558e-4e70-93ec-cd84edfe35e1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6e0203c3-558e-4e70-93ec-cd84edfe35e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:14:12.869246 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.869129    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6e0203c3-558e-4e70-93ec-cd84edfe35e1-config-out\") pod \"prometheus-k8s-0\" (UID: \"6e0203c3-558e-4e70-93ec-cd84edfe35e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:14:12.869854 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.869835    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/6e0203c3-558e-4e70-93ec-cd84edfe35e1-config\") pod \"prometheus-k8s-0\" (UID: \"6e0203c3-558e-4e70-93ec-cd84edfe35e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:14:12.870057 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.870039    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6e0203c3-558e-4e70-93ec-cd84edfe35e1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6e0203c3-558e-4e70-93ec-cd84edfe35e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:14:12.870377 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.870359    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6e0203c3-558e-4e70-93ec-cd84edfe35e1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6e0203c3-558e-4e70-93ec-cd84edfe35e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:14:12.886742 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:12.886721    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz6zk\" (UniqueName: \"kubernetes.io/projected/6e0203c3-558e-4e70-93ec-cd84edfe35e1-kube-api-access-fz6zk\") pod \"prometheus-k8s-0\" (UID: \"6e0203c3-558e-4e70-93ec-cd84edfe35e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:14:13.060453 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:13.060371    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:14:13.406999 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:13.406928    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 22:14:13.409562 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:14:13.409531    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e0203c3_558e_4e70_93ec_cd84edfe35e1.slice/crio-4b37b7694f0237c4351b9f324884e6f702b05643d58ef4d3f53b71bb595a6b96 WatchSource:0}: Error finding container 4b37b7694f0237c4351b9f324884e6f702b05643d58ef4d3f53b71bb595a6b96: Status 404 returned error can't find the container with id 4b37b7694f0237c4351b9f324884e6f702b05643d58ef4d3f53b71bb595a6b96
Nov 03 22:14:13.505681 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:13.505651    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6e0203c3-558e-4e70-93ec-cd84edfe35e1","Type":"ContainerStarted","Data":"4b37b7694f0237c4351b9f324884e6f702b05643d58ef4d3f53b71bb595a6b96"}
Nov 03 22:14:14.085127 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:14.085093    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e33b576-1208-4325-87e9-4351920bfa06" path="/var/lib/kubelet/pods/1e33b576-1208-4325-87e9-4351920bfa06/volumes"
Nov 03 22:14:14.516160 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:14.516110    4431 generic.go:356] "Generic (PLEG): container finished" podID="6e0203c3-558e-4e70-93ec-cd84edfe35e1" containerID="c79f1280a371f899e1e76de368520b207e39f863c6cfbd3e6bbbe9e676e9ec38" exitCode=0
Nov 03 22:14:14.516542 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:14.516185    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6e0203c3-558e-4e70-93ec-cd84edfe35e1","Type":"ContainerDied","Data":"c79f1280a371f899e1e76de368520b207e39f863c6cfbd3e6bbbe9e676e9ec38"}
Nov 03 22:14:15.522796 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:15.522762    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6e0203c3-558e-4e70-93ec-cd84edfe35e1","Type":"ContainerStarted","Data":"085940b70faba4a694dfd9ad91d3ed10079fb9ebc5369040efed643f8e5fa864"}
Nov 03 22:14:15.523187 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:15.522830    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6e0203c3-558e-4e70-93ec-cd84edfe35e1","Type":"ContainerStarted","Data":"d627f722fd29920c660aa9e595093e6277ef737c08eff6f24a3b1eb53c43bb55"}
Nov 03 22:14:15.523187 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:15.522861    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6e0203c3-558e-4e70-93ec-cd84edfe35e1","Type":"ContainerStarted","Data":"3377979c9a350b40b80daed740459a077d0ab4223056f5d05d4bf41570ba6570"}
Nov 03 22:14:15.523187 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:15.522890    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6e0203c3-558e-4e70-93ec-cd84edfe35e1","Type":"ContainerStarted","Data":"e3b9582cf6243b607c0e09f855d6decb71874081984984c5fb0f9e3f6ff05fb0"}
Nov 03 22:14:15.523187 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:15.522922    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6e0203c3-558e-4e70-93ec-cd84edfe35e1","Type":"ContainerStarted","Data":"fd820aafcf69bff5d4f0cb8782893e9b7a0adfee20923ea41dcd87b98d1d2692"}
Nov 03 22:14:15.523187 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:15.522947    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6e0203c3-558e-4e70-93ec-cd84edfe35e1","Type":"ContainerStarted","Data":"34f3a25747e26d4e0b61e245808c6440dd5c742fc91d5ff2d83025c14f27002d"}
Nov 03 22:14:15.591080 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:15.591033    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.567734915 podStartE2EDuration="3.591018372s" totalImagesPullingTime="0s" totalInitContainerRuntime="23.283457ms" isStatefulPod=false podCreationTimestamp="2026-11-03 22:14:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:14:15.590399286 +0000 UTC m=+79.616255159" watchObservedRunningTime="2026-11-03 22:14:15.591018372 +0000 UTC m=+79.616874238"
Nov 03 22:14:16.902090 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:16.902053    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:14:17.038162 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:17.038122    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:14:18.060485 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:18.060447    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:14:31.257590 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:14:31.257552    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:14:32.244483 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:32.244451    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:14:32.383773 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:32.383739    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:14:38.817293 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:38.817260    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:14:38.958061 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:38.958028    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:14:44.006939 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:44.006900    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:14:59.543383 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:59.543347    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:14:59.684409 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:14:59.684375    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:15:01.257231 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:15:01.257170    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:15:04.046355 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:15:04.046315    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:15:13.060532 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:15:13.060451    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:15:13.075922 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:15:13.075885    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:15:13.744664 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:15:13.744636    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 22:15:21.322228 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:15:21.322190    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:15:31.256960 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:15:31.256915    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:15:44.126401 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:15:44.126363    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:16:01.256666 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:16:01.256630    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:16:19.874849 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:16:19.874815    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:16:20.026430 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:16:20.026394    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:16:21.369409 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:16:21.369374    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:16:21.516923 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:16:21.516884    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:16:22.714371 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:16:22.714334    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:16:27.036456 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:16:27.036420    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:16:27.178201 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:16:27.178167    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:16:27.365194 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:16:27.365114    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:16:27.508358 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:16:27.508326    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:16:27.661277 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:16:27.661200    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:16:30.655975 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:16:30.655943    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:16:31.257695 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:16:31.257656    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:16:43.278947 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:16:43.278870    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:16:44.232553 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:16:44.232521    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:16:58.469630 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:16:58.469584    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:16:58.614294 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:16:58.614260    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:17:01.256918 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:17:01.256875    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:17:02.226877 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:02.226843    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:17:02.366658 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:02.366620    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:17:05.498011 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:05.497982    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:17:05.644977 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:05.644942    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:17:08.546576 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:08.546540    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:17:08.689511 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:08.689480    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:17:09.757653 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:09.757595    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:17:09.978634 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:09.978597    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-2trwz"]
Nov 03 22:17:09.980933 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:09.980916    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-2trwz"
Nov 03 22:17:09.990175 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:09.990148    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ff75ee09-9686-4ffc-ba62-6fa8b4fca65c-metrics-cert\") pod \"opendatahub-operator-controller-manager-65cf5586b-2trwz\" (UID: \"ff75ee09-9686-4ffc-ba62-6fa8b4fca65c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-2trwz"
Nov 03 22:17:09.990306 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:09.990195    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ff75ee09-9686-4ffc-ba62-6fa8b4fca65c-manifests\") pod \"opendatahub-operator-controller-manager-65cf5586b-2trwz\" (UID: \"ff75ee09-9686-4ffc-ba62-6fa8b4fca65c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-2trwz"
Nov 03 22:17:09.990306 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:09.990210    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ff75ee09-9686-4ffc-ba62-6fa8b4fca65c-tmp\") pod \"opendatahub-operator-controller-manager-65cf5586b-2trwz\" (UID: \"ff75ee09-9686-4ffc-ba62-6fa8b4fca65c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-2trwz"
Nov 03 22:17:09.990401 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:09.990303    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ff75ee09-9686-4ffc-ba62-6fa8b4fca65c-webhook-cert\") pod \"opendatahub-operator-controller-manager-65cf5586b-2trwz\" (UID: \"ff75ee09-9686-4ffc-ba62-6fa8b4fca65c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-2trwz"
Nov 03 22:17:09.990401 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:09.990339    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ff75ee09-9686-4ffc-ba62-6fa8b4fca65c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-65cf5586b-2trwz\" (UID: \"ff75ee09-9686-4ffc-ba62-6fa8b4fca65c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-2trwz"
Nov 03 22:17:09.990401 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:09.990366    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf4kt\" (UniqueName: \"kubernetes.io/projected/ff75ee09-9686-4ffc-ba62-6fa8b4fca65c-kube-api-access-lf4kt\") pod \"opendatahub-operator-controller-manager-65cf5586b-2trwz\" (UID: \"ff75ee09-9686-4ffc-ba62-6fa8b4fca65c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-2trwz"
Nov 03 22:17:10.025827 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:10.025754    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-2trwz"]
Nov 03 22:17:10.091382 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:10.091355    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ff75ee09-9686-4ffc-ba62-6fa8b4fca65c-webhook-cert\") pod \"opendatahub-operator-controller-manager-65cf5586b-2trwz\" (UID: \"ff75ee09-9686-4ffc-ba62-6fa8b4fca65c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-2trwz"
Nov 03 22:17:10.091550 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:10.091416    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ff75ee09-9686-4ffc-ba62-6fa8b4fca65c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-65cf5586b-2trwz\" (UID: \"ff75ee09-9686-4ffc-ba62-6fa8b4fca65c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-2trwz"
Nov 03 22:17:10.091550 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:10.091465    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf4kt\" (UniqueName: \"kubernetes.io/projected/ff75ee09-9686-4ffc-ba62-6fa8b4fca65c-kube-api-access-lf4kt\") pod \"opendatahub-operator-controller-manager-65cf5586b-2trwz\" (UID: \"ff75ee09-9686-4ffc-ba62-6fa8b4fca65c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-2trwz"
Nov 03 22:17:10.091550 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:10.091521    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ff75ee09-9686-4ffc-ba62-6fa8b4fca65c-metrics-cert\") pod \"opendatahub-operator-controller-manager-65cf5586b-2trwz\" (UID: \"ff75ee09-9686-4ffc-ba62-6fa8b4fca65c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-2trwz"
Nov 03 22:17:10.091716 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:10.091628    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ff75ee09-9686-4ffc-ba62-6fa8b4fca65c-manifests\") pod \"opendatahub-operator-controller-manager-65cf5586b-2trwz\" (UID: \"ff75ee09-9686-4ffc-ba62-6fa8b4fca65c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-2trwz"
Nov 03 22:17:10.091716 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:10.091653    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ff75ee09-9686-4ffc-ba62-6fa8b4fca65c-tmp\") pod \"opendatahub-operator-controller-manager-65cf5586b-2trwz\" (UID: \"ff75ee09-9686-4ffc-ba62-6fa8b4fca65c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-2trwz"
Nov 03 22:17:10.092085 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:10.092060    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ff75ee09-9686-4ffc-ba62-6fa8b4fca65c-manifests\") pod \"opendatahub-operator-controller-manager-65cf5586b-2trwz\" (UID: \"ff75ee09-9686-4ffc-ba62-6fa8b4fca65c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-2trwz"
Nov 03 22:17:10.093820 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:10.093800    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ff75ee09-9686-4ffc-ba62-6fa8b4fca65c-tmp\") pod \"opendatahub-operator-controller-manager-65cf5586b-2trwz\" (UID: \"ff75ee09-9686-4ffc-ba62-6fa8b4fca65c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-2trwz"
Nov 03 22:17:10.097947 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:10.097923    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 03 22:17:10.098238 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:10.098201    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 03 22:17:10.098288 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:10.098233    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-wmvl7\""
Nov 03 22:17:10.098288 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:10.098246    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 03 22:17:10.104506 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:10.104479    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ff75ee09-9686-4ffc-ba62-6fa8b4fca65c-metrics-cert\") pod \"opendatahub-operator-controller-manager-65cf5586b-2trwz\" (UID: \"ff75ee09-9686-4ffc-ba62-6fa8b4fca65c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-2trwz"
Nov 03 22:17:10.104593 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:10.104563    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ff75ee09-9686-4ffc-ba62-6fa8b4fca65c-webhook-cert\") pod \"opendatahub-operator-controller-manager-65cf5586b-2trwz\" (UID: \"ff75ee09-9686-4ffc-ba62-6fa8b4fca65c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-2trwz"
Nov 03 22:17:10.104633 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:10.104590    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ff75ee09-9686-4ffc-ba62-6fa8b4fca65c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-65cf5586b-2trwz\" (UID: \"ff75ee09-9686-4ffc-ba62-6fa8b4fca65c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-2trwz"
Nov 03 22:17:10.105732 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:10.105715    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 03 22:17:10.105843 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:10.105829    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 03 22:17:10.114455 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:10.114433    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf4kt\" (UniqueName: \"kubernetes.io/projected/ff75ee09-9686-4ffc-ba62-6fa8b4fca65c-kube-api-access-lf4kt\") pod \"opendatahub-operator-controller-manager-65cf5586b-2trwz\" (UID: \"ff75ee09-9686-4ffc-ba62-6fa8b4fca65c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-2trwz"
Nov 03 22:17:10.116588 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:10.116567    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:17:10.258942 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:10.258913    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:17:10.293133 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:10.293038    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-2trwz"
Nov 03 22:17:10.404841 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:10.404809    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:17:10.645947 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:10.645888    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-2trwz"]
Nov 03 22:17:10.648285 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:17:10.648257    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff75ee09_9686_4ffc_ba62_6fa8b4fca65c.slice/crio-59cafc3cd8af5745f718358a495962f047709f462fbdbfc4916139dbdf939f19 WatchSource:0}: Error finding container 59cafc3cd8af5745f718358a495962f047709f462fbdbfc4916139dbdf939f19: Status 404 returned error can't find the container with id 59cafc3cd8af5745f718358a495962f047709f462fbdbfc4916139dbdf939f19
Nov 03 22:17:11.136362 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:11.136323    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-2trwz" event={"ID":"ff75ee09-9686-4ffc-ba62-6fa8b4fca65c","Type":"ContainerStarted","Data":"59cafc3cd8af5745f718358a495962f047709f462fbdbfc4916139dbdf939f19"}
Nov 03 22:17:12.564545 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:12.564512    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:17:13.552038 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:13.552008    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:17:13.767012 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:13.766936    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:17:14.149593 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:14.149508    4431 generic.go:356] "Generic (PLEG): container finished" podID="ff75ee09-9686-4ffc-ba62-6fa8b4fca65c" containerID="958bdd8f0b38e9f0e57e3446896eafe7f1a9ecb87089b9e5b49ae6b144cb7a17" exitCode=0
Nov 03 22:17:14.149593 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:14.149569    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-2trwz" event={"ID":"ff75ee09-9686-4ffc-ba62-6fa8b4fca65c","Type":"ContainerDied","Data":"958bdd8f0b38e9f0e57e3446896eafe7f1a9ecb87089b9e5b49ae6b144cb7a17"}
Nov 03 22:17:14.755370 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:14.755284    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:17:15.155723 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:15.155627    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-2trwz" event={"ID":"ff75ee09-9686-4ffc-ba62-6fa8b4fca65c","Type":"ContainerStarted","Data":"ad456ce9e2a47412503c9fe5a39ef40116e1ece69362d7eca8eca34af02adb65"}
Nov 03 22:17:15.156119 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:15.155780    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-2trwz"
Nov 03 22:17:15.247945 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:15.247893    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-2trwz" podStartSLOduration=2.733938007 podStartE2EDuration="6.24788007s" totalImagesPullingTime="3.080756331s" totalInitContainerRuntime="433.185732ms" isStatefulPod=false podCreationTimestamp="2026-11-03 22:17:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:17:15.235414799 +0000 UTC m=+259.261270661" watchObservedRunningTime="2026-11-03 22:17:15.24788007 +0000 UTC m=+259.273735923"
Nov 03 22:17:15.892418 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:15.892379    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:17:19.517729 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:19.517696    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:17:19.661358 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:19.661329    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:17:21.555264 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:21.555211    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:17:23.079668 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:23.079638    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:17:23.238941 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:23.238909    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:17:23.538545 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:23.538513    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:17:25.549081 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:25.549043    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:17:25.896943 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:25.896859    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:17:26.161451 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:26.161431    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-2trwz"
Nov 03 22:17:27.812037 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:27.812006    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-82jtt"]
Nov 03 22:17:27.818412 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:27.818393    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-82jtt"
Nov 03 22:17:27.837881 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:27.837852    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-82jtt"]
Nov 03 22:17:27.848753 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:27.848728    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Nov 03 22:17:27.848881 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:27.848804    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-9zcrl\""
Nov 03 22:17:27.851677 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:27.851657    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Nov 03 22:17:27.851758 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:27.851689    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 03 22:17:27.851758 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:27.851706    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 03 22:17:27.870245 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:27.870191    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:17:27.933655 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:27.933617    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc4vp\" (UniqueName: \"kubernetes.io/projected/8910869a-df66-49d2-a6b0-2a98eda07ea2-kube-api-access-lc4vp\") pod \"kuadrant-console-plugin-66c4fb575d-82jtt\" (UID: \"8910869a-df66-49d2-a6b0-2a98eda07ea2\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-82jtt"
Nov 03 22:17:27.933799 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:27.933667    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8910869a-df66-49d2-a6b0-2a98eda07ea2-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-82jtt\" (UID: \"8910869a-df66-49d2-a6b0-2a98eda07ea2\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-82jtt"
Nov 03 22:17:27.933799 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:27.933685    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/8910869a-df66-49d2-a6b0-2a98eda07ea2-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-82jtt\" (UID: \"8910869a-df66-49d2-a6b0-2a98eda07ea2\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-82jtt"
Nov 03 22:17:28.010731 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:28.010698    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:17:28.035012 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:28.034982    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc4vp\" (UniqueName: \"kubernetes.io/projected/8910869a-df66-49d2-a6b0-2a98eda07ea2-kube-api-access-lc4vp\") pod \"kuadrant-console-plugin-66c4fb575d-82jtt\" (UID: \"8910869a-df66-49d2-a6b0-2a98eda07ea2\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-82jtt"
Nov 03 22:17:28.035124 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:28.035030    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8910869a-df66-49d2-a6b0-2a98eda07ea2-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-82jtt\" (UID: \"8910869a-df66-49d2-a6b0-2a98eda07ea2\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-82jtt"
Nov 03 22:17:28.035124 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:28.035051    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/8910869a-df66-49d2-a6b0-2a98eda07ea2-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-82jtt\" (UID: \"8910869a-df66-49d2-a6b0-2a98eda07ea2\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-82jtt"
Nov 03 22:17:28.035681 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:28.035659    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8910869a-df66-49d2-a6b0-2a98eda07ea2-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-82jtt\" (UID: \"8910869a-df66-49d2-a6b0-2a98eda07ea2\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-82jtt"
Nov 03 22:17:28.037467 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:28.037446    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/8910869a-df66-49d2-a6b0-2a98eda07ea2-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-82jtt\" (UID: \"8910869a-df66-49d2-a6b0-2a98eda07ea2\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-82jtt"
Nov 03 22:17:28.058457 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:28.058425    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc4vp\" (UniqueName: \"kubernetes.io/projected/8910869a-df66-49d2-a6b0-2a98eda07ea2-kube-api-access-lc4vp\") pod \"kuadrant-console-plugin-66c4fb575d-82jtt\" (UID: \"8910869a-df66-49d2-a6b0-2a98eda07ea2\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-82jtt"
Nov 03 22:17:28.148021 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:28.147946    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-82jtt"
Nov 03 22:17:28.451876 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:28.451846    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-82jtt"]
Nov 03 22:17:28.455116 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:17:28.455088    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8910869a_df66_49d2_a6b0_2a98eda07ea2.slice/crio-70fa0c1436b4d4fc513f87769d7eb5baeea8c69dbb78804bd722588cea392677 WatchSource:0}: Error finding container 70fa0c1436b4d4fc513f87769d7eb5baeea8c69dbb78804bd722588cea392677: Status 404 returned error can't find the container with id 70fa0c1436b4d4fc513f87769d7eb5baeea8c69dbb78804bd722588cea392677
Nov 03 22:17:28.750189 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:28.750117    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:17:28.967658 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:28.967622    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:17:29.208678 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:29.208633    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-82jtt" event={"ID":"8910869a-df66-49d2-a6b0-2a98eda07ea2","Type":"ContainerStarted","Data":"70fa0c1436b4d4fc513f87769d7eb5baeea8c69dbb78804bd722588cea392677"}
Nov 03 22:17:29.345769 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:29.345721    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:17:29.567559 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:29.567476    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:17:31.260778 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:17:31.260741    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:17:31.455786 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:31.455743    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:17:32.246507 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:32.246469    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:17:32.415510 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:32.415469    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:17:32.503161 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:32.503089    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-mrr4d"]
Nov 03 22:17:32.505184 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:32.505167    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-mrr4d"
Nov 03 22:17:32.530122 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:32.530083    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Nov 03 22:17:32.542199 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:32.542172    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Nov 03 22:17:32.542556 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:32.542536    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-7rxlt\""
Nov 03 22:17:32.543713 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:32.543690    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-mrr4d"]
Nov 03 22:17:32.574130 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:32.574094    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvdf2\" (UniqueName: \"kubernetes.io/projected/85a4d227-3a7e-4270-aabe-3a79d7f51c1d-kube-api-access-tvdf2\") pod \"cert-manager-webhook-5dbbbf75cc-mrr4d\" (UID: \"85a4d227-3a7e-4270-aabe-3a79d7f51c1d\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-mrr4d"
Nov 03 22:17:32.574309 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:32.574147    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/85a4d227-3a7e-4270-aabe-3a79d7f51c1d-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-mrr4d\" (UID: \"85a4d227-3a7e-4270-aabe-3a79d7f51c1d\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-mrr4d"
Nov 03 22:17:32.674744 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:32.674705    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvdf2\" (UniqueName: \"kubernetes.io/projected/85a4d227-3a7e-4270-aabe-3a79d7f51c1d-kube-api-access-tvdf2\") pod \"cert-manager-webhook-5dbbbf75cc-mrr4d\" (UID: \"85a4d227-3a7e-4270-aabe-3a79d7f51c1d\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-mrr4d"
Nov 03 22:17:32.674920 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:32.674765    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/85a4d227-3a7e-4270-aabe-3a79d7f51c1d-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-mrr4d\" (UID: \"85a4d227-3a7e-4270-aabe-3a79d7f51c1d\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-mrr4d"
Nov 03 22:17:32.698425 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:32.698360    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/85a4d227-3a7e-4270-aabe-3a79d7f51c1d-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-mrr4d\" (UID: \"85a4d227-3a7e-4270-aabe-3a79d7f51c1d\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-mrr4d"
Nov 03 22:17:32.698588 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:32.698431    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvdf2\" (UniqueName: \"kubernetes.io/projected/85a4d227-3a7e-4270-aabe-3a79d7f51c1d-kube-api-access-tvdf2\") pod \"cert-manager-webhook-5dbbbf75cc-mrr4d\" (UID: \"85a4d227-3a7e-4270-aabe-3a79d7f51c1d\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-mrr4d"
Nov 03 22:17:32.817608 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:32.817520    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-mrr4d"
Nov 03 22:17:33.128079 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:33.128051    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-mrr4d"]
Nov 03 22:17:33.130798 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:17:33.130762    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod85a4d227_3a7e_4270_aabe_3a79d7f51c1d.slice/crio-0cb1bef79a8c6cb2ceb5b41b36bc3c918dffd3890cf15d3e41418dfc843275c8 WatchSource:0}: Error finding container 0cb1bef79a8c6cb2ceb5b41b36bc3c918dffd3890cf15d3e41418dfc843275c8: Status 404 returned error can't find the container with id 0cb1bef79a8c6cb2ceb5b41b36bc3c918dffd3890cf15d3e41418dfc843275c8
Nov 03 22:17:33.226385 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:33.226350    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-82jtt" event={"ID":"8910869a-df66-49d2-a6b0-2a98eda07ea2","Type":"ContainerStarted","Data":"a11a13c443cc12c469ed20a729777a80bf2f515bcdfec5c52c975148d3379fe6"}
Nov 03 22:17:33.227568 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:33.227538    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-mrr4d" event={"ID":"85a4d227-3a7e-4270-aabe-3a79d7f51c1d","Type":"ContainerStarted","Data":"0cb1bef79a8c6cb2ceb5b41b36bc3c918dffd3890cf15d3e41418dfc843275c8"}
Nov 03 22:17:33.267665 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:33.267614    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-82jtt" podStartSLOduration=1.7190133410000001 podStartE2EDuration="6.267600681s" totalImagesPullingTime="4.54858734s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:17:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:17:33.266044323 +0000 UTC m=+277.291900186" watchObservedRunningTime="2026-11-03 22:17:33.267600681 +0000 UTC m=+277.293456543"
Nov 03 22:17:33.609416 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:33.609381    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:17:34.559746 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:34.559534    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:17:34.649294 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:34.649255    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-z5qcx"]
Nov 03 22:17:34.651827 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:34.651803    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-z5qcx"
Nov 03 22:17:34.670422 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:34.670383    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-vclcm\""
Nov 03 22:17:34.684400 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:34.684369    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-z5qcx"]
Nov 03 22:17:34.694581 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:34.694553    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a36de2dc-0c08-4277-8964-f610642e3bf2-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-z5qcx\" (UID: \"a36de2dc-0c08-4277-8964-f610642e3bf2\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-z5qcx"
Nov 03 22:17:34.694717 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:34.694662    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgps8\" (UniqueName: \"kubernetes.io/projected/a36de2dc-0c08-4277-8964-f610642e3bf2-kube-api-access-jgps8\") pod \"cert-manager-cainjector-5c57bbd4fc-z5qcx\" (UID: \"a36de2dc-0c08-4277-8964-f610642e3bf2\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-z5qcx"
Nov 03 22:17:34.796129 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:34.796089    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a36de2dc-0c08-4277-8964-f610642e3bf2-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-z5qcx\" (UID: \"a36de2dc-0c08-4277-8964-f610642e3bf2\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-z5qcx"
Nov 03 22:17:34.796337 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:34.796159    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgps8\" (UniqueName: \"kubernetes.io/projected/a36de2dc-0c08-4277-8964-f610642e3bf2-kube-api-access-jgps8\") pod \"cert-manager-cainjector-5c57bbd4fc-z5qcx\" (UID: \"a36de2dc-0c08-4277-8964-f610642e3bf2\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-z5qcx"
Nov 03 22:17:34.828960 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:34.828893    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a36de2dc-0c08-4277-8964-f610642e3bf2-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-z5qcx\" (UID: \"a36de2dc-0c08-4277-8964-f610642e3bf2\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-z5qcx"
Nov 03 22:17:34.836135 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:34.835887    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgps8\" (UniqueName: \"kubernetes.io/projected/a36de2dc-0c08-4277-8964-f610642e3bf2-kube-api-access-jgps8\") pod \"cert-manager-cainjector-5c57bbd4fc-z5qcx\" (UID: \"a36de2dc-0c08-4277-8964-f610642e3bf2\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-z5qcx"
Nov 03 22:17:34.871171 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:34.871134    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:17:34.964106 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:34.964069    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-z5qcx"
Nov 03 22:17:35.282361 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:35.282330    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-z5qcx"]
Nov 03 22:17:35.285551 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:17:35.285520    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda36de2dc_0c08_4277_8964_f610642e3bf2.slice/crio-d5f81a2923ae5d1d8a96c95b23d2025e51fc5278d68221f3c10a467f12ac0b0f WatchSource:0}: Error finding container d5f81a2923ae5d1d8a96c95b23d2025e51fc5278d68221f3c10a467f12ac0b0f: Status 404 returned error can't find the container with id d5f81a2923ae5d1d8a96c95b23d2025e51fc5278d68221f3c10a467f12ac0b0f
Nov 03 22:17:36.248713 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:36.248680    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-z5qcx" event={"ID":"a36de2dc-0c08-4277-8964-f610642e3bf2","Type":"ContainerStarted","Data":"d5f81a2923ae5d1d8a96c95b23d2025e51fc5278d68221f3c10a467f12ac0b0f"}
Nov 03 22:17:36.974998 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:36.974938    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:17:37.115754 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:37.115717    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:17:37.254420 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:37.254384    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-mrr4d" event={"ID":"85a4d227-3a7e-4270-aabe-3a79d7f51c1d","Type":"ContainerStarted","Data":"1c3c8bd8bd62b9308496fb736f76b28bae28c799db9bf47efd6c77f4e2d67065"}
Nov 03 22:17:37.254734 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:37.254519    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-mrr4d"
Nov 03 22:17:37.295202 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:37.295154    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-mrr4d" podStartSLOduration=2.054060264 podStartE2EDuration="5.29514052s" totalImagesPullingTime="3.241080256s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:17:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:17:37.294292656 +0000 UTC m=+281.320148520" watchObservedRunningTime="2026-11-03 22:17:37.29514052 +0000 UTC m=+281.320996382"
Nov 03 22:17:37.344921 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:37.344879    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:17:37.494774 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:37.494652    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:17:37.813762 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:37.813663    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:17:38.017414 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:38.017378    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:17:38.260031 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:38.259995    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-z5qcx" event={"ID":"a36de2dc-0c08-4277-8964-f610642e3bf2","Type":"ContainerStarted","Data":"0e68134e202949c4f291cfbe8fcb99b534e650948173b90ad6f63d0cf2838c42"}
Nov 03 22:17:38.302327 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:38.302278    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-z5qcx" podStartSLOduration=2.364082451 podStartE2EDuration="4.302263055s" totalImagesPullingTime="1.938180604s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:17:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:17:38.301674963 +0000 UTC m=+282.327530822" watchObservedRunningTime="2026-11-03 22:17:38.302263055 +0000 UTC m=+282.328118917"
Nov 03 22:17:42.332337 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:42.332302    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:17:42.634463 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:42.634386    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:17:42.787802 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:42.787769    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-tfzx5"]
Nov 03 22:17:42.791416 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:42.791395    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-tfzx5"
Nov 03 22:17:42.804343 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:42.804316    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:17:42.807527 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:42.807508    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-24mpf\""
Nov 03 22:17:42.821643 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:42.821617    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-tfzx5"]
Nov 03 22:17:42.865287 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:42.865241    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8972ac1a-d480-4a83-b5f1-044b20a5067f-bound-sa-token\") pod \"cert-manager-597b95cb84-tfzx5\" (UID: \"8972ac1a-d480-4a83-b5f1-044b20a5067f\") " pod="cert-manager/cert-manager-597b95cb84-tfzx5"
Nov 03 22:17:42.865456 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:42.865298    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqn6z\" (UniqueName: \"kubernetes.io/projected/8972ac1a-d480-4a83-b5f1-044b20a5067f-kube-api-access-tqn6z\") pod \"cert-manager-597b95cb84-tfzx5\" (UID: \"8972ac1a-d480-4a83-b5f1-044b20a5067f\") " pod="cert-manager/cert-manager-597b95cb84-tfzx5"
Nov 03 22:17:42.966465 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:42.966420    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8972ac1a-d480-4a83-b5f1-044b20a5067f-bound-sa-token\") pod \"cert-manager-597b95cb84-tfzx5\" (UID: \"8972ac1a-d480-4a83-b5f1-044b20a5067f\") " pod="cert-manager/cert-manager-597b95cb84-tfzx5"
Nov 03 22:17:42.966628 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:42.966489    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqn6z\" (UniqueName: \"kubernetes.io/projected/8972ac1a-d480-4a83-b5f1-044b20a5067f-kube-api-access-tqn6z\") pod \"cert-manager-597b95cb84-tfzx5\" (UID: \"8972ac1a-d480-4a83-b5f1-044b20a5067f\") " pod="cert-manager/cert-manager-597b95cb84-tfzx5"
Nov 03 22:17:43.001239 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:43.001192    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8972ac1a-d480-4a83-b5f1-044b20a5067f-bound-sa-token\") pod \"cert-manager-597b95cb84-tfzx5\" (UID: \"8972ac1a-d480-4a83-b5f1-044b20a5067f\") " pod="cert-manager/cert-manager-597b95cb84-tfzx5"
Nov 03 22:17:43.001385 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:43.001255    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqn6z\" (UniqueName: \"kubernetes.io/projected/8972ac1a-d480-4a83-b5f1-044b20a5067f-kube-api-access-tqn6z\") pod \"cert-manager-597b95cb84-tfzx5\" (UID: \"8972ac1a-d480-4a83-b5f1-044b20a5067f\") " pod="cert-manager/cert-manager-597b95cb84-tfzx5"
Nov 03 22:17:43.007918 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:43.007888    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:17:43.050858 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:43.050824    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6b75ff7d64-48pqm"]
Nov 03 22:17:43.054094 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:43.054074    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6b75ff7d64-48pqm"
Nov 03 22:17:43.076812 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:43.076785    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 03 22:17:43.077025 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:43.077004    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 03 22:17:43.087002 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:43.086972    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6b75ff7d64-48pqm"]
Nov 03 22:17:43.102950 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:43.102927    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-tfzx5"
Nov 03 22:17:43.152610 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:43.152546    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Nov 03 22:17:43.153657 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:43.153000    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Nov 03 22:17:43.153860 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:43.153242    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-qjnck\""
Nov 03 22:17:43.167892 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:43.167844    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z99c9\" (UniqueName: \"kubernetes.io/projected/970868dd-c8fb-4960-a494-d76890bc0ff6-kube-api-access-z99c9\") pod \"kube-auth-proxy-6b75ff7d64-48pqm\" (UID: \"970868dd-c8fb-4960-a494-d76890bc0ff6\") " pod="openshift-ingress/kube-auth-proxy-6b75ff7d64-48pqm"
Nov 03 22:17:43.168033 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:43.167961    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/970868dd-c8fb-4960-a494-d76890bc0ff6-tmp\") pod \"kube-auth-proxy-6b75ff7d64-48pqm\" (UID: \"970868dd-c8fb-4960-a494-d76890bc0ff6\") " pod="openshift-ingress/kube-auth-proxy-6b75ff7d64-48pqm"
Nov 03 22:17:43.168033 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:43.168009    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/970868dd-c8fb-4960-a494-d76890bc0ff6-tls-certs\") pod \"kube-auth-proxy-6b75ff7d64-48pqm\" (UID: \"970868dd-c8fb-4960-a494-d76890bc0ff6\") " pod="openshift-ingress/kube-auth-proxy-6b75ff7d64-48pqm"
Nov 03 22:17:43.206616 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:43.206583    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:17:43.262257 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:43.262171    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-mrr4d"
Nov 03 22:17:43.268889 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:43.268856    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/970868dd-c8fb-4960-a494-d76890bc0ff6-tmp\") pod \"kube-auth-proxy-6b75ff7d64-48pqm\" (UID: \"970868dd-c8fb-4960-a494-d76890bc0ff6\") " pod="openshift-ingress/kube-auth-proxy-6b75ff7d64-48pqm"
Nov 03 22:17:43.268979 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:43.268907    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/970868dd-c8fb-4960-a494-d76890bc0ff6-tls-certs\") pod \"kube-auth-proxy-6b75ff7d64-48pqm\" (UID: \"970868dd-c8fb-4960-a494-d76890bc0ff6\") " pod="openshift-ingress/kube-auth-proxy-6b75ff7d64-48pqm"
Nov 03 22:17:43.268979 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:43.268951    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z99c9\" (UniqueName: \"kubernetes.io/projected/970868dd-c8fb-4960-a494-d76890bc0ff6-kube-api-access-z99c9\") pod \"kube-auth-proxy-6b75ff7d64-48pqm\" (UID: \"970868dd-c8fb-4960-a494-d76890bc0ff6\") " pod="openshift-ingress/kube-auth-proxy-6b75ff7d64-48pqm"
Nov 03 22:17:43.271134 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:43.271103    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/970868dd-c8fb-4960-a494-d76890bc0ff6-tmp\") pod \"kube-auth-proxy-6b75ff7d64-48pqm\" (UID: \"970868dd-c8fb-4960-a494-d76890bc0ff6\") " pod="openshift-ingress/kube-auth-proxy-6b75ff7d64-48pqm"
Nov 03 22:17:43.271278 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:43.271260    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/970868dd-c8fb-4960-a494-d76890bc0ff6-tls-certs\") pod \"kube-auth-proxy-6b75ff7d64-48pqm\" (UID: \"970868dd-c8fb-4960-a494-d76890bc0ff6\") " pod="openshift-ingress/kube-auth-proxy-6b75ff7d64-48pqm"
Nov 03 22:17:43.305893 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:43.305858    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z99c9\" (UniqueName: \"kubernetes.io/projected/970868dd-c8fb-4960-a494-d76890bc0ff6-kube-api-access-z99c9\") pod \"kube-auth-proxy-6b75ff7d64-48pqm\" (UID: \"970868dd-c8fb-4960-a494-d76890bc0ff6\") " pod="openshift-ingress/kube-auth-proxy-6b75ff7d64-48pqm"
Nov 03 22:17:43.372571 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:43.372537    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6b75ff7d64-48pqm"
Nov 03 22:17:43.388244 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:43.388199    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:17:43.412435 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:43.412409    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-tfzx5"]
Nov 03 22:17:43.415676 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:17:43.415588    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8972ac1a_d480_4a83_b5f1_044b20a5067f.slice/crio-47cca6f23c35b00169b792e113e2a482273ddd95ef5a13013c928dbda2ab54f7 WatchSource:0}: Error finding container 47cca6f23c35b00169b792e113e2a482273ddd95ef5a13013c928dbda2ab54f7: Status 404 returned error can't find the container with id 47cca6f23c35b00169b792e113e2a482273ddd95ef5a13013c928dbda2ab54f7
Nov 03 22:17:43.576659 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:43.576559    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:17:43.676513 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:43.676485    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6b75ff7d64-48pqm"]
Nov 03 22:17:43.679608 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:17:43.679582    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod970868dd_c8fb_4960_a494_d76890bc0ff6.slice/crio-5f1f6fb3c1adabb54a0b405166b5c6fde4ba13cad16901c9a024905feb94b1bd WatchSource:0}: Error finding container 5f1f6fb3c1adabb54a0b405166b5c6fde4ba13cad16901c9a024905feb94b1bd: Status 404 returned error can't find the container with id 5f1f6fb3c1adabb54a0b405166b5c6fde4ba13cad16901c9a024905feb94b1bd
Nov 03 22:17:44.287377 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:44.287331    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-tfzx5" event={"ID":"8972ac1a-d480-4a83-b5f1-044b20a5067f","Type":"ContainerStarted","Data":"00ad81eae8eb18fa537e1f6f0a485a403f687be90b8d0b534572d7d04b411fe0"}
Nov 03 22:17:44.287581 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:44.287393    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-tfzx5" event={"ID":"8972ac1a-d480-4a83-b5f1-044b20a5067f","Type":"ContainerStarted","Data":"47cca6f23c35b00169b792e113e2a482273ddd95ef5a13013c928dbda2ab54f7"}
Nov 03 22:17:44.290160 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:44.290129    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6b75ff7d64-48pqm" event={"ID":"970868dd-c8fb-4960-a494-d76890bc0ff6","Type":"ContainerStarted","Data":"5f1f6fb3c1adabb54a0b405166b5c6fde4ba13cad16901c9a024905feb94b1bd"}
Nov 03 22:17:44.340023 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:44.339961    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-tfzx5" podStartSLOduration=2.339947153 podStartE2EDuration="2.339947153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:17:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:17:44.33825562 +0000 UTC m=+288.364111481" watchObservedRunningTime="2026-11-03 22:17:44.339947153 +0000 UTC m=+288.365803015"
Nov 03 22:17:44.383323 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:44.383280    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:17:44.534985 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:44.534953    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:17:44.927951 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:44.927917    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:17:45.965605 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:45.965570    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:17:46.300725 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:46.300641    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6b75ff7d64-48pqm" event={"ID":"970868dd-c8fb-4960-a494-d76890bc0ff6","Type":"ContainerStarted","Data":"2f1c346c0d7b2b8dbdac1de4d9e4d1fd32a7252bd1bb862854b7a02bf4858c0f"}
Nov 03 22:17:46.343106 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:46.343042    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6b75ff7d64-48pqm" podStartSLOduration=1.021684094 podStartE2EDuration="3.343024368s" totalImagesPullingTime="2.321340274s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:17:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:17:46.341889792 +0000 UTC m=+290.367745656" watchObservedRunningTime="2026-11-03 22:17:46.343024368 +0000 UTC m=+290.368880228"
Nov 03 22:17:48.410807 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:48.410772    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:17:48.651867 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:48.651830    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:17:53.192928 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:53.192897    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:17:53.333098 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:53.333057    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:17:55.802659 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:55.802621    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:17:55.962504 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:55.962468    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:17:56.109529 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:56.109453    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mx5q8_ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea/ovn-acl-logging/0.log"
Nov 03 22:17:56.110443 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:56.110419    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mx5q8_ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea/ovn-acl-logging/0.log"
Nov 03 22:17:58.135683 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:58.135644    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:17:58.298232 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:17:58.298187    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:18:00.004614 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:00.004579    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:18:00.152995 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:00.152958    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:18:01.257878 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:18:01.257842    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:18:04.599266 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:04.599207    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:18:04.741474 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:04.741441    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:18:09.403263 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:09.403205    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:18:09.575362 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:09.575329    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:18:10.403086 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:10.403016    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-kg4dw"]
Nov 03 22:18:10.413607 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:10.413581    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-kg4dw"
Nov 03 22:18:10.430025 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:10.429991    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Nov 03 22:18:10.442512 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:10.442485    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-kg4dw"]
Nov 03 22:18:10.444616 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:10.444584    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:18:10.494315 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:10.494279    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-kg4dw"]
Nov 03 22:18:10.539563 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:10.539515    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3d488e3b-0050-476c-b084-01095a223d76-config-file\") pod \"limitador-limitador-74dfcfb894-kg4dw\" (UID: \"3d488e3b-0050-476c-b084-01095a223d76\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-kg4dw"
Nov 03 22:18:10.539726 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:10.539579    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf8n6\" (UniqueName: \"kubernetes.io/projected/3d488e3b-0050-476c-b084-01095a223d76-kube-api-access-zf8n6\") pod \"limitador-limitador-74dfcfb894-kg4dw\" (UID: \"3d488e3b-0050-476c-b084-01095a223d76\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-kg4dw"
Nov 03 22:18:10.587707 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:10.587673    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:18:10.640581 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:10.640547    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3d488e3b-0050-476c-b084-01095a223d76-config-file\") pod \"limitador-limitador-74dfcfb894-kg4dw\" (UID: \"3d488e3b-0050-476c-b084-01095a223d76\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-kg4dw"
Nov 03 22:18:10.640581 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:10.640594    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf8n6\" (UniqueName: \"kubernetes.io/projected/3d488e3b-0050-476c-b084-01095a223d76-kube-api-access-zf8n6\") pod \"limitador-limitador-74dfcfb894-kg4dw\" (UID: \"3d488e3b-0050-476c-b084-01095a223d76\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-kg4dw"
Nov 03 22:18:10.641327 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:10.641304    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3d488e3b-0050-476c-b084-01095a223d76-config-file\") pod \"limitador-limitador-74dfcfb894-kg4dw\" (UID: \"3d488e3b-0050-476c-b084-01095a223d76\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-kg4dw"
Nov 03 22:18:10.666007 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:10.665953    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf8n6\" (UniqueName: \"kubernetes.io/projected/3d488e3b-0050-476c-b084-01095a223d76-kube-api-access-zf8n6\") pod \"limitador-limitador-74dfcfb894-kg4dw\" (UID: \"3d488e3b-0050-476c-b084-01095a223d76\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-kg4dw"
Nov 03 22:18:10.734996 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:10.734964    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-kg4dw"
Nov 03 22:18:11.053484 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:11.053336    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-kg4dw"]
Nov 03 22:18:11.056177 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:18:11.056146    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d488e3b_0050_476c_b084_01095a223d76.slice/crio-3240f7a33d6c5e891809f26bb30d1eaa4aacd403aa20b2f72930a7229bc811f4 WatchSource:0}: Error finding container 3240f7a33d6c5e891809f26bb30d1eaa4aacd403aa20b2f72930a7229bc811f4: Status 404 returned error can't find the container with id 3240f7a33d6c5e891809f26bb30d1eaa4aacd403aa20b2f72930a7229bc811f4
Nov 03 22:18:11.058024 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:11.058006    4431 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 22:18:11.336932 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:11.336842    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:18:11.406174 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:11.406141    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-kg4dw" event={"ID":"3d488e3b-0050-476c-b084-01095a223d76","Type":"ContainerStarted","Data":"3240f7a33d6c5e891809f26bb30d1eaa4aacd403aa20b2f72930a7229bc811f4"}
Nov 03 22:18:11.482856 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:11.482822    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:18:13.421357 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:13.421324    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-kg4dw" event={"ID":"3d488e3b-0050-476c-b084-01095a223d76","Type":"ContainerStarted","Data":"723948de9a217eae8a98e926d23a6db779a698bbdd3c30e5c05f0fb563261c58"}
Nov 03 22:18:13.421747 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:13.421445    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-kg4dw"
Nov 03 22:18:13.463138 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:13.463084    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-kg4dw" podStartSLOduration=1.283911565 podStartE2EDuration="3.46307003s" totalImagesPullingTime="2.179158465s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:18:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:18:13.461450795 +0000 UTC m=+317.487306659" watchObservedRunningTime="2026-11-03 22:18:13.46307003 +0000 UTC m=+317.488925891"
Nov 03 22:18:14.130900 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:14.130868    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:18:14.292992 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:14.292959    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:18:19.330519 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:19.330485    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:18:19.472346 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:19.472313    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:18:19.722392 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:19.722351    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:18:19.875257 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:19.875204    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:18:20.037441 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:20.037351    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:18:24.426555 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:24.426521    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-kg4dw"
Nov 03 22:18:25.996204 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:25.996155    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:18:26.137149 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:26.137119    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:18:27.072465 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:27.072434    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-kg4dw"]
Nov 03 22:18:27.072858 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:27.072665    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-kg4dw" podUID="3d488e3b-0050-476c-b084-01095a223d76" containerName="limitador" containerID="cri-o://723948de9a217eae8a98e926d23a6db779a698bbdd3c30e5c05f0fb563261c58" gracePeriod=30
Nov 03 22:18:27.621169 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:27.621144    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-kg4dw"
Nov 03 22:18:27.693509 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:27.693474    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d488e3b-0050-476c-b084-01095a223d76-config-file\" (UniqueName: \"kubernetes.io/configmap/3d488e3b-0050-476c-b084-01095a223d76-config-file\") pod \"3d488e3b-0050-476c-b084-01095a223d76\" (UID: \"3d488e3b-0050-476c-b084-01095a223d76\") "
Nov 03 22:18:27.693679 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:27.693581    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d488e3b-0050-476c-b084-01095a223d76-kube-api-access-zf8n6\" (UniqueName: \"kubernetes.io/projected/3d488e3b-0050-476c-b084-01095a223d76-kube-api-access-zf8n6\") pod \"3d488e3b-0050-476c-b084-01095a223d76\" (UID: \"3d488e3b-0050-476c-b084-01095a223d76\") "
Nov 03 22:18:27.693828 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:27.693806    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d488e3b-0050-476c-b084-01095a223d76-config-file" pod "3d488e3b-0050-476c-b084-01095a223d76" (UID: "3d488e3b-0050-476c-b084-01095a223d76"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:18:27.695770 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:27.695741    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d488e3b-0050-476c-b084-01095a223d76-kube-api-access-zf8n6" pod "3d488e3b-0050-476c-b084-01095a223d76" (UID: "3d488e3b-0050-476c-b084-01095a223d76"). InnerVolumeSpecName "kube-api-access-zf8n6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:18:27.794122 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:27.794092    4431 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3d488e3b-0050-476c-b084-01095a223d76-config-file\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:18:27.794122 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:27.794118    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zf8n6\" (UniqueName: \"kubernetes.io/projected/3d488e3b-0050-476c-b084-01095a223d76-kube-api-access-zf8n6\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:18:28.483332 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:28.483294    4431 generic.go:356] "Generic (PLEG): container finished" podID="3d488e3b-0050-476c-b084-01095a223d76" containerID="723948de9a217eae8a98e926d23a6db779a698bbdd3c30e5c05f0fb563261c58" exitCode=0
Nov 03 22:18:28.483771 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:28.483344    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-kg4dw"
Nov 03 22:18:28.483771 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:28.483378    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-kg4dw" event={"ID":"3d488e3b-0050-476c-b084-01095a223d76","Type":"ContainerDied","Data":"723948de9a217eae8a98e926d23a6db779a698bbdd3c30e5c05f0fb563261c58"}
Nov 03 22:18:28.483771 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:28.483438    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-kg4dw" event={"ID":"3d488e3b-0050-476c-b084-01095a223d76","Type":"ContainerDied","Data":"3240f7a33d6c5e891809f26bb30d1eaa4aacd403aa20b2f72930a7229bc811f4"}
Nov 03 22:18:28.483771 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:28.483461    4431 scope.go:122] "RemoveContainer" containerID="723948de9a217eae8a98e926d23a6db779a698bbdd3c30e5c05f0fb563261c58"
Nov 03 22:18:28.491310 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:28.491295    4431 scope.go:122] "RemoveContainer" containerID="723948de9a217eae8a98e926d23a6db779a698bbdd3c30e5c05f0fb563261c58"
Nov 03 22:18:28.491585 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:18:28.491568    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"723948de9a217eae8a98e926d23a6db779a698bbdd3c30e5c05f0fb563261c58\": container with ID starting with 723948de9a217eae8a98e926d23a6db779a698bbdd3c30e5c05f0fb563261c58 not found: ID does not exist" containerID="723948de9a217eae8a98e926d23a6db779a698bbdd3c30e5c05f0fb563261c58"
Nov 03 22:18:28.491635 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:28.491593    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"723948de9a217eae8a98e926d23a6db779a698bbdd3c30e5c05f0fb563261c58"} err="failed to get container status \"723948de9a217eae8a98e926d23a6db779a698bbdd3c30e5c05f0fb563261c58\": rpc error: code = NotFound desc = could not find container \"723948de9a217eae8a98e926d23a6db779a698bbdd3c30e5c05f0fb563261c58\": container with ID starting with 723948de9a217eae8a98e926d23a6db779a698bbdd3c30e5c05f0fb563261c58 not found: ID does not exist"
Nov 03 22:18:28.534265 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:28.534208    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-kg4dw"]
Nov 03 22:18:28.546334 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:28.546306    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-kg4dw"]
Nov 03 22:18:30.085004 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:30.084970    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d488e3b-0050-476c-b084-01095a223d76" path="/var/lib/kubelet/pods/3d488e3b-0050-476c-b084-01095a223d76/volumes"
Nov 03 22:18:30.734880 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:30.734845    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:18:30.874825 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:30.874792    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:18:31.257642 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:18:31.257602    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:18:31.670187 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:31.670154    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:18:35.573534 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:35.573498    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:18:35.716048 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:35.716010    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:18:44.493392 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:44.493351    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:18:44.654516 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:44.654482    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:18:44.794474 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:44.794397    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:18:44.928792 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:44.928762    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-h52bb"]
Nov 03 22:18:44.929124 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:44.929104    4431 cpu_manager.go:414] "containerMap: removing container" podUID="3d488e3b-0050-476c-b084-01095a223d76" containerName="limitador"
Nov 03 22:18:44.929124 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:44.929122    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d488e3b-0050-476c-b084-01095a223d76" containerName="limitador"
Nov 03 22:18:44.929338 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:44.929196    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d488e3b-0050-476c-b084-01095a223d76" containerName="limitador"
Nov 03 22:18:44.934103 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:44.934082    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-h52bb"
Nov 03 22:18:44.945101 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:44.945074    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:18:44.951829 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:44.951794    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 03 22:18:44.964021 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:44.963991    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-m9js5\""
Nov 03 22:18:44.964264 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:44.963993    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 03 22:18:44.964376 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:44.964093    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Nov 03 22:18:44.966305 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:44.966279    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-h52bb"]
Nov 03 22:18:44.967198 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:44.967180    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-4zggb"]
Nov 03 22:18:44.972292 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:44.972274    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4zggb"
Nov 03 22:18:44.985125 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:44.985101    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-4zggb"]
Nov 03 22:18:45.010910 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:45.010880    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-rtdd4"]
Nov 03 22:18:45.014420 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:45.014403    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-rtdd4"
Nov 03 22:18:45.033928 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:45.033901    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-rtdd4"]
Nov 03 22:18:45.044028 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:45.043988    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9b824440-4f91-46d4-beb5-a081da92aa4d-cert\") pod \"kserve-controller-manager-776cc66d47-h52bb\" (UID: \"9b824440-4f91-46d4-beb5-a081da92aa4d\") " pod="opendatahub/kserve-controller-manager-776cc66d47-h52bb"
Nov 03 22:18:45.044174 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:45.044061    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2hrf\" (UniqueName: \"kubernetes.io/projected/c811b17a-e08a-43e2-8986-59991e1a9804-kube-api-access-n2hrf\") pod \"llmisvc-controller-manager-869d5dcc5d-4zggb\" (UID: \"c811b17a-e08a-43e2-8986-59991e1a9804\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4zggb"
Nov 03 22:18:45.044174 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:45.044131    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl8pj\" (UniqueName: \"kubernetes.io/projected/f1b5a649-dc8d-48f5-9308-5a3370e5afcf-kube-api-access-tl8pj\") pod \"model-serving-api-75bd89fdfc-rtdd4\" (UID: \"f1b5a649-dc8d-48f5-9308-5a3370e5afcf\") " pod="opendatahub/model-serving-api-75bd89fdfc-rtdd4"
Nov 03 22:18:45.044373 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:45.044179    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr5bg\" (UniqueName: \"kubernetes.io/projected/9b824440-4f91-46d4-beb5-a081da92aa4d-kube-api-access-xr5bg\") pod \"kserve-controller-manager-776cc66d47-h52bb\" (UID: \"9b824440-4f91-46d4-beb5-a081da92aa4d\") " pod="opendatahub/kserve-controller-manager-776cc66d47-h52bb"
Nov 03 22:18:45.044373 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:45.044200    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1b5a649-dc8d-48f5-9308-5a3370e5afcf-tls-certs\") pod \"model-serving-api-75bd89fdfc-rtdd4\" (UID: \"f1b5a649-dc8d-48f5-9308-5a3370e5afcf\") " pod="opendatahub/model-serving-api-75bd89fdfc-rtdd4"
Nov 03 22:18:45.044373 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:45.044311    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c811b17a-e08a-43e2-8986-59991e1a9804-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-4zggb\" (UID: \"c811b17a-e08a-43e2-8986-59991e1a9804\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4zggb"
Nov 03 22:18:45.051928 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:45.051863    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Nov 03 22:18:45.052058 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:45.052036    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-lz8nk\""
Nov 03 22:18:45.052123 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:45.052059    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Nov 03 22:18:45.052123 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:45.052077    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-6pnpb\""
Nov 03 22:18:45.066624 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:45.066601    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-sj75c"]
Nov 03 22:18:45.070123 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:45.070107    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-sj75c"
Nov 03 22:18:45.089246 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:45.089207    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-sj75c"]
Nov 03 22:18:45.096423 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:45.096395    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:18:45.120207 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:45.120178    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Nov 03 22:18:45.120405 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:45.120256    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-pt4bg\""
Nov 03 22:18:45.120515 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:45.120499    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\""
Nov 03 22:18:45.145587 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:45.145545    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c811b17a-e08a-43e2-8986-59991e1a9804-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-4zggb\" (UID: \"c811b17a-e08a-43e2-8986-59991e1a9804\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4zggb"
Nov 03 22:18:45.145733 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:45.145595    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkf79\" (UniqueName: \"kubernetes.io/projected/0d02a958-18da-44d7-a774-29e918540e52-kube-api-access-hkf79\") pod \"odh-model-controller-769875db56-sj75c\" (UID: \"0d02a958-18da-44d7-a774-29e918540e52\") " pod="opendatahub/odh-model-controller-769875db56-sj75c"
Nov 03 22:18:45.145733 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:45.145636    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d02a958-18da-44d7-a774-29e918540e52-metrics-certs\") pod \"odh-model-controller-769875db56-sj75c\" (UID: \"0d02a958-18da-44d7-a774-29e918540e52\") " pod="opendatahub/odh-model-controller-769875db56-sj75c"
Nov 03 22:18:45.145733 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:45.145669    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9b824440-4f91-46d4-beb5-a081da92aa4d-cert\") pod \"kserve-controller-manager-776cc66d47-h52bb\" (UID: \"9b824440-4f91-46d4-beb5-a081da92aa4d\") " pod="opendatahub/kserve-controller-manager-776cc66d47-h52bb"
Nov 03 22:18:45.145908 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:45.145740    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2hrf\" (UniqueName: \"kubernetes.io/projected/c811b17a-e08a-43e2-8986-59991e1a9804-kube-api-access-n2hrf\") pod \"llmisvc-controller-manager-869d5dcc5d-4zggb\" (UID: \"c811b17a-e08a-43e2-8986-59991e1a9804\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4zggb"
Nov 03 22:18:45.145908 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:45.145842    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl8pj\" (UniqueName: \"kubernetes.io/projected/f1b5a649-dc8d-48f5-9308-5a3370e5afcf-kube-api-access-tl8pj\") pod \"model-serving-api-75bd89fdfc-rtdd4\" (UID: \"f1b5a649-dc8d-48f5-9308-5a3370e5afcf\") " pod="opendatahub/model-serving-api-75bd89fdfc-rtdd4"
Nov 03 22:18:45.145908 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:45.145873    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr5bg\" (UniqueName: \"kubernetes.io/projected/9b824440-4f91-46d4-beb5-a081da92aa4d-kube-api-access-xr5bg\") pod \"kserve-controller-manager-776cc66d47-h52bb\" (UID: \"9b824440-4f91-46d4-beb5-a081da92aa4d\") " pod="opendatahub/kserve-controller-manager-776cc66d47-h52bb"
Nov 03 22:18:45.146055 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:45.145905    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1b5a649-dc8d-48f5-9308-5a3370e5afcf-tls-certs\") pod \"model-serving-api-75bd89fdfc-rtdd4\" (UID: \"f1b5a649-dc8d-48f5-9308-5a3370e5afcf\") " pod="opendatahub/model-serving-api-75bd89fdfc-rtdd4"
Nov 03 22:18:45.146055 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:45.145933    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0d02a958-18da-44d7-a774-29e918540e52-cert\") pod \"odh-model-controller-769875db56-sj75c\" (UID: \"0d02a958-18da-44d7-a774-29e918540e52\") " pod="opendatahub/odh-model-controller-769875db56-sj75c"
Nov 03 22:18:45.148179 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:45.148153    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c811b17a-e08a-43e2-8986-59991e1a9804-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-4zggb\" (UID: \"c811b17a-e08a-43e2-8986-59991e1a9804\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4zggb"
Nov 03 22:18:45.148286 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:45.148160    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1b5a649-dc8d-48f5-9308-5a3370e5afcf-tls-certs\") pod \"model-serving-api-75bd89fdfc-rtdd4\" (UID: \"f1b5a649-dc8d-48f5-9308-5a3370e5afcf\") " pod="opendatahub/model-serving-api-75bd89fdfc-rtdd4"
Nov 03 22:18:45.148286 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:45.148163    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9b824440-4f91-46d4-beb5-a081da92aa4d-cert\") pod \"kserve-controller-manager-776cc66d47-h52bb\" (UID: \"9b824440-4f91-46d4-beb5-a081da92aa4d\") " pod="opendatahub/kserve-controller-manager-776cc66d47-h52bb"
Nov 03 22:18:45.169534 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:45.169511    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr5bg\" (UniqueName: \"kubernetes.io/projected/9b824440-4f91-46d4-beb5-a081da92aa4d-kube-api-access-xr5bg\") pod \"kserve-controller-manager-776cc66d47-h52bb\" (UID: \"9b824440-4f91-46d4-beb5-a081da92aa4d\") " pod="opendatahub/kserve-controller-manager-776cc66d47-h52bb"
Nov 03 22:18:45.169645 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:45.169537    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2hrf\" (UniqueName: \"kubernetes.io/projected/c811b17a-e08a-43e2-8986-59991e1a9804-kube-api-access-n2hrf\") pod \"llmisvc-controller-manager-869d5dcc5d-4zggb\" (UID: \"c811b17a-e08a-43e2-8986-59991e1a9804\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4zggb"
Nov 03 22:18:45.170059 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:45.170038    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl8pj\" (UniqueName: \"kubernetes.io/projected/f1b5a649-dc8d-48f5-9308-5a3370e5afcf-kube-api-access-tl8pj\") pod \"model-serving-api-75bd89fdfc-rtdd4\" (UID: \"f1b5a649-dc8d-48f5-9308-5a3370e5afcf\") " pod="opendatahub/model-serving-api-75bd89fdfc-rtdd4"
Nov 03 22:18:45.244872 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:45.244839    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:18:45.247054 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:45.247028    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0d02a958-18da-44d7-a774-29e918540e52-cert\") pod \"odh-model-controller-769875db56-sj75c\" (UID: \"0d02a958-18da-44d7-a774-29e918540e52\") " pod="opendatahub/odh-model-controller-769875db56-sj75c"
Nov 03 22:18:45.247211 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:45.247089    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkf79\" (UniqueName: \"kubernetes.io/projected/0d02a958-18da-44d7-a774-29e918540e52-kube-api-access-hkf79\") pod \"odh-model-controller-769875db56-sj75c\" (UID: \"0d02a958-18da-44d7-a774-29e918540e52\") " pod="opendatahub/odh-model-controller-769875db56-sj75c"
Nov 03 22:18:45.247211 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:45.247115    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d02a958-18da-44d7-a774-29e918540e52-metrics-certs\") pod \"odh-model-controller-769875db56-sj75c\" (UID: \"0d02a958-18da-44d7-a774-29e918540e52\") " pod="opendatahub/odh-model-controller-769875db56-sj75c"
Nov 03 22:18:45.247756 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:45.247552    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-h52bb"
Nov 03 22:18:45.249698 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:45.249668    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d02a958-18da-44d7-a774-29e918540e52-metrics-certs\") pod \"odh-model-controller-769875db56-sj75c\" (UID: \"0d02a958-18da-44d7-a774-29e918540e52\") " pod="opendatahub/odh-model-controller-769875db56-sj75c"
Nov 03 22:18:45.249797 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:45.249697    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0d02a958-18da-44d7-a774-29e918540e52-cert\") pod \"odh-model-controller-769875db56-sj75c\" (UID: \"0d02a958-18da-44d7-a774-29e918540e52\") " pod="opendatahub/odh-model-controller-769875db56-sj75c"
Nov 03 22:18:45.270210 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:45.270185    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkf79\" (UniqueName: \"kubernetes.io/projected/0d02a958-18da-44d7-a774-29e918540e52-kube-api-access-hkf79\") pod \"odh-model-controller-769875db56-sj75c\" (UID: \"0d02a958-18da-44d7-a774-29e918540e52\") " pod="opendatahub/odh-model-controller-769875db56-sj75c"
Nov 03 22:18:45.286052 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:45.286000    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4zggb"
Nov 03 22:18:45.329290 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:45.327535    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-rtdd4"
Nov 03 22:18:45.383318 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:45.383282    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-sj75c"
Nov 03 22:18:45.544245 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:45.544198    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-h52bb"]
Nov 03 22:18:45.547211 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:18:45.547182    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b824440_4f91_46d4_beb5_a081da92aa4d.slice/crio-eab9ff1b4a4f6e0a0b3865763b78367e995b0b0ab104646901cbc45bdd75d32a WatchSource:0}: Error finding container eab9ff1b4a4f6e0a0b3865763b78367e995b0b0ab104646901cbc45bdd75d32a: Status 404 returned error can't find the container with id eab9ff1b4a4f6e0a0b3865763b78367e995b0b0ab104646901cbc45bdd75d32a
Nov 03 22:18:45.554718 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:45.554695    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:18:45.556177 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:45.556155    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-h52bb" event={"ID":"9b824440-4f91-46d4-beb5-a081da92aa4d","Type":"ContainerStarted","Data":"eab9ff1b4a4f6e0a0b3865763b78367e995b0b0ab104646901cbc45bdd75d32a"}
Nov 03 22:18:45.642327 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:45.642303    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-4zggb"]
Nov 03 22:18:45.645282 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:18:45.645250    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc811b17a_e08a_43e2_8986_59991e1a9804.slice/crio-be74d2fe6f591ada7d6b732a2cac54c68cdd404e5c918f90f3aa9bdc24000ebe WatchSource:0}: Error finding container be74d2fe6f591ada7d6b732a2cac54c68cdd404e5c918f90f3aa9bdc24000ebe: Status 404 returned error can't find the container with id be74d2fe6f591ada7d6b732a2cac54c68cdd404e5c918f90f3aa9bdc24000ebe
Nov 03 22:18:45.681014 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:45.680992    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-rtdd4"]
Nov 03 22:18:45.683065 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:18:45.683040    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1b5a649_dc8d_48f5_9308_5a3370e5afcf.slice/crio-c2f9f5019a852abaea94b215955745e0c7ce0ba1e01cae1fcd0099c1b8e78859 WatchSource:0}: Error finding container c2f9f5019a852abaea94b215955745e0c7ce0ba1e01cae1fcd0099c1b8e78859: Status 404 returned error can't find the container with id c2f9f5019a852abaea94b215955745e0c7ce0ba1e01cae1fcd0099c1b8e78859
Nov 03 22:18:45.695007 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:45.694979    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-sj75c"]
Nov 03 22:18:45.696721 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:18:45.696696    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d02a958_18da_44d7_a774_29e918540e52.slice/crio-022aab0dbcb99d987b25735b21494aefa95fb0c8c39cb34f4ce10d817c5200d2 WatchSource:0}: Error finding container 022aab0dbcb99d987b25735b21494aefa95fb0c8c39cb34f4ce10d817c5200d2: Status 404 returned error can't find the container with id 022aab0dbcb99d987b25735b21494aefa95fb0c8c39cb34f4ce10d817c5200d2
Nov 03 22:18:46.566352 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:46.566191    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4zggb" event={"ID":"c811b17a-e08a-43e2-8986-59991e1a9804","Type":"ContainerStarted","Data":"be74d2fe6f591ada7d6b732a2cac54c68cdd404e5c918f90f3aa9bdc24000ebe"}
Nov 03 22:18:46.570239 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:46.570134    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-rtdd4" event={"ID":"f1b5a649-dc8d-48f5-9308-5a3370e5afcf","Type":"ContainerStarted","Data":"c2f9f5019a852abaea94b215955745e0c7ce0ba1e01cae1fcd0099c1b8e78859"}
Nov 03 22:18:46.571712 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:46.571640    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-sj75c" event={"ID":"0d02a958-18da-44d7-a774-29e918540e52","Type":"ContainerStarted","Data":"022aab0dbcb99d987b25735b21494aefa95fb0c8c39cb34f4ce10d817c5200d2"}
Nov 03 22:18:47.374894 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:47.374844    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:18:47.517673 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:47.517612    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:18:47.655202 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:47.655071    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:18:47.793436 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:47.793384    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:18:48.585326 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:48.585283    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-rtdd4" event={"ID":"f1b5a649-dc8d-48f5-9308-5a3370e5afcf","Type":"ContainerStarted","Data":"b3d5a5ed637d89ff1a3d7042e0f3878449a2ec373dc80a3d40dbc133c1e8741c"}
Nov 03 22:18:48.585983 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:48.585960    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-rtdd4"
Nov 03 22:18:48.631296 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:48.630390    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-rtdd4" podStartSLOduration=1.939297957 podStartE2EDuration="4.630368665s" totalImagesPullingTime="2.691070708s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:18:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:18:48.629017617 +0000 UTC m=+352.654873481" watchObservedRunningTime="2026-11-03 22:18:48.630368665 +0000 UTC m=+352.656224530"
Nov 03 22:18:49.590924 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:49.590871    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4zggb" event={"ID":"c811b17a-e08a-43e2-8986-59991e1a9804","Type":"ContainerStarted","Data":"de1c73759fc74bfb1161059f1a522fba019508e9d93cbe6ec18556d9aff2cef6"}
Nov 03 22:18:49.591426 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:49.590984    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4zggb"
Nov 03 22:18:49.592653 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:49.592589    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-h52bb" event={"ID":"9b824440-4f91-46d4-beb5-a081da92aa4d","Type":"ContainerStarted","Data":"b3f43853e6d1ef9c69046a9f8ca254e5e5c353a2049e24942b5a81d80743c89e"}
Nov 03 22:18:49.592653 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:49.592652    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-h52bb"
Nov 03 22:18:49.594292 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:49.594262    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-sj75c" event={"ID":"0d02a958-18da-44d7-a774-29e918540e52","Type":"ContainerStarted","Data":"38295c87addc363d466eb2c1ef02414913bd2e5cde54227bb67411cec23f6a5f"}
Nov 03 22:18:49.594410 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:49.594340    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-sj75c"
Nov 03 22:18:49.631615 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:49.631509    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4zggb" podStartSLOduration=2.520970824 podStartE2EDuration="5.63149438s" totalImagesPullingTime="3.110523556s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:18:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:18:49.630375905 +0000 UTC m=+353.656231767" watchObservedRunningTime="2026-11-03 22:18:49.63149438 +0000 UTC m=+353.657350296"
Nov 03 22:18:49.670622 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:49.670570    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-h52bb" podStartSLOduration=2.453244999 podStartE2EDuration="5.670553848s" totalImagesPullingTime="3.217308849s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:18:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:18:49.670241844 +0000 UTC m=+353.696097707" watchObservedRunningTime="2026-11-03 22:18:49.670553848 +0000 UTC m=+353.696409711"
Nov 03 22:18:49.710448 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:49.710391    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-sj75c" podStartSLOduration=1.054181194 podStartE2EDuration="4.710376446s" totalImagesPullingTime="3.656195252s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:18:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:18:49.70992317 +0000 UTC m=+353.735779036" watchObservedRunningTime="2026-11-03 22:18:49.710376446 +0000 UTC m=+353.736232314"
Nov 03 22:18:50.777374 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:50.777339    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:18:50.915287 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:50.915243    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:18:51.058955 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:51.058870    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:18:54.509691 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:54.509648    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:18:54.655104 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:54.655058    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:18:55.336700 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:55.336663    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:18:55.476648 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:55.476601    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:18:57.290976 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:57.290782    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kdpwc"]
Nov 03 22:18:57.295697 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:57.295675    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kdpwc"
Nov 03 22:18:57.314151 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:57.314122    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kdpwc"]
Nov 03 22:18:57.323827 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:57.323798    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\""
Nov 03 22:18:57.323988 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:57.323856    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\""
Nov 03 22:18:57.349656 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:57.349626    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\""
Nov 03 22:18:57.349804 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:57.349697    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-rlph9\""
Nov 03 22:18:57.367132 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:57.367100    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/202b2e09-563a-4ff1-bff1-edb9c7cc3d18-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-kdpwc\" (UID: \"202b2e09-563a-4ff1-bff1-edb9c7cc3d18\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kdpwc"
Nov 03 22:18:57.367310 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:57.367150    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/202b2e09-563a-4ff1-bff1-edb9c7cc3d18-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-kdpwc\" (UID: \"202b2e09-563a-4ff1-bff1-edb9c7cc3d18\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kdpwc"
Nov 03 22:18:57.367310 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:57.367288    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/202b2e09-563a-4ff1-bff1-edb9c7cc3d18-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-kdpwc\" (UID: \"202b2e09-563a-4ff1-bff1-edb9c7cc3d18\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kdpwc"
Nov 03 22:18:57.367432 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:57.367412    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv4zs\" (UniqueName: \"kubernetes.io/projected/202b2e09-563a-4ff1-bff1-edb9c7cc3d18-kube-api-access-vv4zs\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-kdpwc\" (UID: \"202b2e09-563a-4ff1-bff1-edb9c7cc3d18\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kdpwc"
Nov 03 22:18:57.416384 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:57.416353    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:18:57.468848 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:57.468807    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/202b2e09-563a-4ff1-bff1-edb9c7cc3d18-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-kdpwc\" (UID: \"202b2e09-563a-4ff1-bff1-edb9c7cc3d18\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kdpwc"
Nov 03 22:18:57.469020 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:57.468872    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/202b2e09-563a-4ff1-bff1-edb9c7cc3d18-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-kdpwc\" (UID: \"202b2e09-563a-4ff1-bff1-edb9c7cc3d18\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kdpwc"
Nov 03 22:18:57.469020 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:57.468948    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv4zs\" (UniqueName: \"kubernetes.io/projected/202b2e09-563a-4ff1-bff1-edb9c7cc3d18-kube-api-access-vv4zs\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-kdpwc\" (UID: \"202b2e09-563a-4ff1-bff1-edb9c7cc3d18\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kdpwc"
Nov 03 22:18:57.469139 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:57.469021    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/202b2e09-563a-4ff1-bff1-edb9c7cc3d18-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-kdpwc\" (UID: \"202b2e09-563a-4ff1-bff1-edb9c7cc3d18\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kdpwc"
Nov 03 22:18:57.469764 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:57.469741    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/202b2e09-563a-4ff1-bff1-edb9c7cc3d18-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-kdpwc\" (UID: \"202b2e09-563a-4ff1-bff1-edb9c7cc3d18\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kdpwc"
Nov 03 22:18:57.470094 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:57.470072    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/202b2e09-563a-4ff1-bff1-edb9c7cc3d18-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-kdpwc\" (UID: \"202b2e09-563a-4ff1-bff1-edb9c7cc3d18\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kdpwc"
Nov 03 22:18:57.471816 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:57.471789    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/202b2e09-563a-4ff1-bff1-edb9c7cc3d18-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-kdpwc\" (UID: \"202b2e09-563a-4ff1-bff1-edb9c7cc3d18\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kdpwc"
Nov 03 22:18:57.497892 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:57.497860    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv4zs\" (UniqueName: \"kubernetes.io/projected/202b2e09-563a-4ff1-bff1-edb9c7cc3d18-kube-api-access-vv4zs\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-kdpwc\" (UID: \"202b2e09-563a-4ff1-bff1-edb9c7cc3d18\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kdpwc"
Nov 03 22:18:57.577574 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:57.577494    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:18:57.611113 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:57.611081    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kdpwc"
Nov 03 22:18:57.915210 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:57.915182    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kdpwc"]
Nov 03 22:18:57.917236 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:18:57.917183    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod202b2e09_563a_4ff1_bff1_edb9c7cc3d18.slice/crio-fecec3e0381453ae0b02fc06ef55694954765224bd4493dd6437674d116ee4ee WatchSource:0}: Error finding container fecec3e0381453ae0b02fc06ef55694954765224bd4493dd6437674d116ee4ee: Status 404 returned error can't find the container with id fecec3e0381453ae0b02fc06ef55694954765224bd4493dd6437674d116ee4ee
Nov 03 22:18:58.641583 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:18:58.641548    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kdpwc" event={"ID":"202b2e09-563a-4ff1-bff1-edb9c7cc3d18","Type":"ContainerStarted","Data":"fecec3e0381453ae0b02fc06ef55694954765224bd4493dd6437674d116ee4ee"}
Nov 03 22:19:00.607389 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:00.607355    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-sj75c"
Nov 03 22:19:00.611244 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:00.611049    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-rtdd4"
Nov 03 22:19:01.264513 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:19:01.264475    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:19:09.696805 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:09.696766    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kdpwc" event={"ID":"202b2e09-563a-4ff1-bff1-edb9c7cc3d18","Type":"ContainerStarted","Data":"bef5f2205ca0c15ce9d149200405eafe800aeaa112230a00d56ecf02892850a7"}
Nov 03 22:19:09.697302 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:09.696901    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kdpwc"
Nov 03 22:19:09.738273 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:09.738209    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kdpwc" podStartSLOduration=1.8359857210000001 podStartE2EDuration="12.738195442s" totalImagesPullingTime="10.902209721s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:18:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:19:09.737409899 +0000 UTC m=+373.763265764" watchObservedRunningTime="2026-11-03 22:19:09.738195442 +0000 UTC m=+373.764051303"
Nov 03 22:19:20.607526 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:20.607494    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4zggb"
Nov 03 22:19:20.610650 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:20.610627    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-h52bb"
Nov 03 22:19:20.708255 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:20.708224    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kdpwc"
Nov 03 22:19:24.078580 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:24.078544    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:19:24.220070 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:24.220037    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:19:25.570936 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:25.570900    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:19:25.721593 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:25.721555    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:19:25.769548 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:25.769517    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-7dpdk"]
Nov 03 22:19:25.836456 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:25.836385    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-7dpdk"]
Nov 03 22:19:25.836596 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:25.836503    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7dpdk"
Nov 03 22:19:25.861848 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:25.861821    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:19:25.863411 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:25.863391    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Nov 03 22:19:25.863519 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:25.863414    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-qn4p2\""
Nov 03 22:19:25.863560 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:25.863531    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 03 22:19:25.863731 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:25.863720    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 22:19:25.863780 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:25.863719    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 22:19:25.922346 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:25.922312    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34e13faa-3065-4742-8ed4-08e657dc7790-openshift-service-ca\") pod \"maas-api-6d669c7fcd-7dpdk\" (UID: \"34e13faa-3065-4742-8ed4-08e657dc7790\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7dpdk"
Nov 03 22:19:25.922494 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:25.922372    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65rkh\" (UniqueName: \"kubernetes.io/projected/34e13faa-3065-4742-8ed4-08e657dc7790-kube-api-access-65rkh\") pod \"maas-api-6d669c7fcd-7dpdk\" (UID: \"34e13faa-3065-4742-8ed4-08e657dc7790\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7dpdk"
Nov 03 22:19:25.922543 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:25.922487    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34e13faa-3065-4742-8ed4-08e657dc7790-maas-api-tls\") pod \"maas-api-6d669c7fcd-7dpdk\" (UID: \"34e13faa-3065-4742-8ed4-08e657dc7790\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7dpdk"
Nov 03 22:19:25.922543 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:25.922511    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34e13faa-3065-4742-8ed4-08e657dc7790-metrics-certs\") pod \"maas-api-6d669c7fcd-7dpdk\" (UID: \"34e13faa-3065-4742-8ed4-08e657dc7790\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7dpdk"
Nov 03 22:19:26.023430 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:26.023385    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65rkh\" (UniqueName: \"kubernetes.io/projected/34e13faa-3065-4742-8ed4-08e657dc7790-kube-api-access-65rkh\") pod \"maas-api-6d669c7fcd-7dpdk\" (UID: \"34e13faa-3065-4742-8ed4-08e657dc7790\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7dpdk"
Nov 03 22:19:26.023607 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:26.023448    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34e13faa-3065-4742-8ed4-08e657dc7790-maas-api-tls\") pod \"maas-api-6d669c7fcd-7dpdk\" (UID: \"34e13faa-3065-4742-8ed4-08e657dc7790\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7dpdk"
Nov 03 22:19:26.023607 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:26.023467    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34e13faa-3065-4742-8ed4-08e657dc7790-metrics-certs\") pod \"maas-api-6d669c7fcd-7dpdk\" (UID: \"34e13faa-3065-4742-8ed4-08e657dc7790\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7dpdk"
Nov 03 22:19:26.023607 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:26.023509    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34e13faa-3065-4742-8ed4-08e657dc7790-openshift-service-ca\") pod \"maas-api-6d669c7fcd-7dpdk\" (UID: \"34e13faa-3065-4742-8ed4-08e657dc7790\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7dpdk"
Nov 03 22:19:26.024132 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:26.024108    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34e13faa-3065-4742-8ed4-08e657dc7790-openshift-service-ca\") pod \"maas-api-6d669c7fcd-7dpdk\" (UID: \"34e13faa-3065-4742-8ed4-08e657dc7790\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7dpdk"
Nov 03 22:19:26.025934 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:26.025910    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34e13faa-3065-4742-8ed4-08e657dc7790-metrics-certs\") pod \"maas-api-6d669c7fcd-7dpdk\" (UID: \"34e13faa-3065-4742-8ed4-08e657dc7790\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7dpdk"
Nov 03 22:19:26.026031 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:26.025960    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34e13faa-3065-4742-8ed4-08e657dc7790-maas-api-tls\") pod \"maas-api-6d669c7fcd-7dpdk\" (UID: \"34e13faa-3065-4742-8ed4-08e657dc7790\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7dpdk"
Nov 03 22:19:26.046968 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:26.046942    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65rkh\" (UniqueName: \"kubernetes.io/projected/34e13faa-3065-4742-8ed4-08e657dc7790-kube-api-access-65rkh\") pod \"maas-api-6d669c7fcd-7dpdk\" (UID: \"34e13faa-3065-4742-8ed4-08e657dc7790\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7dpdk"
Nov 03 22:19:26.148273 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:26.148179    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7dpdk"
Nov 03 22:19:26.446683 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:26.446655    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-7dpdk"]
Nov 03 22:19:26.449158 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:19:26.449132    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34e13faa_3065_4742_8ed4_08e657dc7790.slice/crio-5da6893867ff74207ba4e6d216ad1fd13f13e3156690c54bfe5d84717ec0c036 WatchSource:0}: Error finding container 5da6893867ff74207ba4e6d216ad1fd13f13e3156690c54bfe5d84717ec0c036: Status 404 returned error can't find the container with id 5da6893867ff74207ba4e6d216ad1fd13f13e3156690c54bfe5d84717ec0c036
Nov 03 22:19:26.771387 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:26.771311    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7dpdk" event={"ID":"34e13faa-3065-4742-8ed4-08e657dc7790","Type":"ContainerStarted","Data":"5da6893867ff74207ba4e6d216ad1fd13f13e3156690c54bfe5d84717ec0c036"}
Nov 03 22:19:28.781471 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:28.781436    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7dpdk" event={"ID":"34e13faa-3065-4742-8ed4-08e657dc7790","Type":"ContainerStarted","Data":"5bc62380a4e4190acb8c75492e7e22c54fe1fe8abfe87207451e8fa7e97eef07"}
Nov 03 22:19:28.781883 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:28.781581    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7dpdk"
Nov 03 22:19:28.823690 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:28.823639    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7dpdk" podStartSLOduration=1.647604894 podStartE2EDuration="3.823626799s" totalImagesPullingTime="2.176021905s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:19:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:19:28.82301916 +0000 UTC m=+392.848875035" watchObservedRunningTime="2026-11-03 22:19:28.823626799 +0000 UTC m=+392.849482662"
Nov 03 22:19:31.258106 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:19:31.258069    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:19:34.791947 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:34.791920    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7dpdk"
Nov 03 22:19:35.292451 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:35.292421    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:19:35.506444 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:35.506409    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:19:35.754373 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:35.754333    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-84f7fbd778-pzqvs"]
Nov 03 22:19:35.756994 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:35.756968    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-84f7fbd778-pzqvs"
Nov 03 22:19:35.807724 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:35.807684    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a974d9ab-19cd-448f-99ba-dbfe4446c562-maas-api-tls\") pod \"maas-api-84f7fbd778-pzqvs\" (UID: \"a974d9ab-19cd-448f-99ba-dbfe4446c562\") " pod="odh-ai-gateway-infra/maas-api-84f7fbd778-pzqvs"
Nov 03 22:19:35.808203 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:35.807763    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a974d9ab-19cd-448f-99ba-dbfe4446c562-metrics-certs\") pod \"maas-api-84f7fbd778-pzqvs\" (UID: \"a974d9ab-19cd-448f-99ba-dbfe4446c562\") " pod="odh-ai-gateway-infra/maas-api-84f7fbd778-pzqvs"
Nov 03 22:19:35.808203 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:35.807803    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a974d9ab-19cd-448f-99ba-dbfe4446c562-openshift-service-ca\") pod \"maas-api-84f7fbd778-pzqvs\" (UID: \"a974d9ab-19cd-448f-99ba-dbfe4446c562\") " pod="odh-ai-gateway-infra/maas-api-84f7fbd778-pzqvs"
Nov 03 22:19:35.808203 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:35.808023    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m4td\" (UniqueName: \"kubernetes.io/projected/a974d9ab-19cd-448f-99ba-dbfe4446c562-kube-api-access-2m4td\") pod \"maas-api-84f7fbd778-pzqvs\" (UID: \"a974d9ab-19cd-448f-99ba-dbfe4446c562\") " pod="odh-ai-gateway-infra/maas-api-84f7fbd778-pzqvs"
Nov 03 22:19:35.812378 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:35.812346    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-84f7fbd778-pzqvs"]
Nov 03 22:19:35.908872 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:35.908833    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a974d9ab-19cd-448f-99ba-dbfe4446c562-metrics-certs\") pod \"maas-api-84f7fbd778-pzqvs\" (UID: \"a974d9ab-19cd-448f-99ba-dbfe4446c562\") " pod="odh-ai-gateway-infra/maas-api-84f7fbd778-pzqvs"
Nov 03 22:19:35.908872 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:35.908876    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a974d9ab-19cd-448f-99ba-dbfe4446c562-openshift-service-ca\") pod \"maas-api-84f7fbd778-pzqvs\" (UID: \"a974d9ab-19cd-448f-99ba-dbfe4446c562\") " pod="odh-ai-gateway-infra/maas-api-84f7fbd778-pzqvs"
Nov 03 22:19:35.909122 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:35.909007    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m4td\" (UniqueName: \"kubernetes.io/projected/a974d9ab-19cd-448f-99ba-dbfe4446c562-kube-api-access-2m4td\") pod \"maas-api-84f7fbd778-pzqvs\" (UID: \"a974d9ab-19cd-448f-99ba-dbfe4446c562\") " pod="odh-ai-gateway-infra/maas-api-84f7fbd778-pzqvs"
Nov 03 22:19:35.909122 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:35.909084    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a974d9ab-19cd-448f-99ba-dbfe4446c562-maas-api-tls\") pod \"maas-api-84f7fbd778-pzqvs\" (UID: \"a974d9ab-19cd-448f-99ba-dbfe4446c562\") " pod="odh-ai-gateway-infra/maas-api-84f7fbd778-pzqvs"
Nov 03 22:19:35.909598 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:35.909571    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a974d9ab-19cd-448f-99ba-dbfe4446c562-openshift-service-ca\") pod \"maas-api-84f7fbd778-pzqvs\" (UID: \"a974d9ab-19cd-448f-99ba-dbfe4446c562\") " pod="odh-ai-gateway-infra/maas-api-84f7fbd778-pzqvs"
Nov 03 22:19:35.911422 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:35.911403    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a974d9ab-19cd-448f-99ba-dbfe4446c562-metrics-certs\") pod \"maas-api-84f7fbd778-pzqvs\" (UID: \"a974d9ab-19cd-448f-99ba-dbfe4446c562\") " pod="odh-ai-gateway-infra/maas-api-84f7fbd778-pzqvs"
Nov 03 22:19:35.911606 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:35.911584    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a974d9ab-19cd-448f-99ba-dbfe4446c562-maas-api-tls\") pod \"maas-api-84f7fbd778-pzqvs\" (UID: \"a974d9ab-19cd-448f-99ba-dbfe4446c562\") " pod="odh-ai-gateway-infra/maas-api-84f7fbd778-pzqvs"
Nov 03 22:19:35.933312 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:35.933286    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m4td\" (UniqueName: \"kubernetes.io/projected/a974d9ab-19cd-448f-99ba-dbfe4446c562-kube-api-access-2m4td\") pod \"maas-api-84f7fbd778-pzqvs\" (UID: \"a974d9ab-19cd-448f-99ba-dbfe4446c562\") " pod="odh-ai-gateway-infra/maas-api-84f7fbd778-pzqvs"
Nov 03 22:19:36.068938 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:36.068850    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-84f7fbd778-pzqvs"
Nov 03 22:19:36.364753 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:36.364726    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-84f7fbd778-pzqvs"]
Nov 03 22:19:36.368819 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:19:36.368792    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda974d9ab_19cd_448f_99ba_dbfe4446c562.slice/crio-aabb256be130eee78e5cd97c24a7b08ec0b7ba3ab8f8a7a3e1744316dc8250b2 WatchSource:0}: Error finding container aabb256be130eee78e5cd97c24a7b08ec0b7ba3ab8f8a7a3e1744316dc8250b2: Status 404 returned error can't find the container with id aabb256be130eee78e5cd97c24a7b08ec0b7ba3ab8f8a7a3e1744316dc8250b2
Nov 03 22:19:36.818981 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:36.818947    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-84f7fbd778-pzqvs" event={"ID":"a974d9ab-19cd-448f-99ba-dbfe4446c562","Type":"ContainerStarted","Data":"0914caca3a0d3ac97f522e2c1ecbb2d7a3f1ef33ea1fa8b8e3672c905679fb9a"}
Nov 03 22:19:36.819387 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:36.819000    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-84f7fbd778-pzqvs" event={"ID":"a974d9ab-19cd-448f-99ba-dbfe4446c562","Type":"ContainerStarted","Data":"aabb256be130eee78e5cd97c24a7b08ec0b7ba3ab8f8a7a3e1744316dc8250b2"}
Nov 03 22:19:36.819387 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:36.819041    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-84f7fbd778-pzqvs"
Nov 03 22:19:36.862884 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:36.862836    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-84f7fbd778-pzqvs" podStartSLOduration=1.5682362109999999 podStartE2EDuration="1.862821146s" totalImagesPullingTime="294.584935ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:19:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:19:36.861545454 +0000 UTC m=+400.887401315" watchObservedRunningTime="2026-11-03 22:19:36.862821146 +0000 UTC m=+400.888677045"
Nov 03 22:19:42.828466 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:42.828387    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-84f7fbd778-pzqvs"
Nov 03 22:19:42.904353 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:42.904317    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-7dpdk"]
Nov 03 22:19:42.904562 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:42.904519    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7dpdk" podUID="34e13faa-3065-4742-8ed4-08e657dc7790" containerName="maas-api" containerID="cri-o://5bc62380a4e4190acb8c75492e7e22c54fe1fe8abfe87207451e8fa7e97eef07" gracePeriod=30
Nov 03 22:19:43.147942 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:43.147917    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7dpdk"
Nov 03 22:19:43.170104 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:43.170075    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34e13faa-3065-4742-8ed4-08e657dc7790-metrics-certs\" (UniqueName: \"kubernetes.io/secret/34e13faa-3065-4742-8ed4-08e657dc7790-metrics-certs\") pod \"34e13faa-3065-4742-8ed4-08e657dc7790\" (UID: \"34e13faa-3065-4742-8ed4-08e657dc7790\") "
Nov 03 22:19:43.170265 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:43.170121    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34e13faa-3065-4742-8ed4-08e657dc7790-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34e13faa-3065-4742-8ed4-08e657dc7790-maas-api-tls\") pod \"34e13faa-3065-4742-8ed4-08e657dc7790\" (UID: \"34e13faa-3065-4742-8ed4-08e657dc7790\") "
Nov 03 22:19:43.170265 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:43.170171    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34e13faa-3065-4742-8ed4-08e657dc7790-kube-api-access-65rkh\" (UniqueName: \"kubernetes.io/projected/34e13faa-3065-4742-8ed4-08e657dc7790-kube-api-access-65rkh\") pod \"34e13faa-3065-4742-8ed4-08e657dc7790\" (UID: \"34e13faa-3065-4742-8ed4-08e657dc7790\") "
Nov 03 22:19:43.170396 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:43.170331    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34e13faa-3065-4742-8ed4-08e657dc7790-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34e13faa-3065-4742-8ed4-08e657dc7790-openshift-service-ca\") pod \"34e13faa-3065-4742-8ed4-08e657dc7790\" (UID: \"34e13faa-3065-4742-8ed4-08e657dc7790\") "
Nov 03 22:19:43.170685 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:43.170657    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34e13faa-3065-4742-8ed4-08e657dc7790-openshift-service-ca" pod "34e13faa-3065-4742-8ed4-08e657dc7790" (UID: "34e13faa-3065-4742-8ed4-08e657dc7790"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:19:43.172451 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:43.172422    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34e13faa-3065-4742-8ed4-08e657dc7790-maas-api-tls" pod "34e13faa-3065-4742-8ed4-08e657dc7790" (UID: "34e13faa-3065-4742-8ed4-08e657dc7790"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:19:43.172547 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:43.172455    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34e13faa-3065-4742-8ed4-08e657dc7790-metrics-certs" pod "34e13faa-3065-4742-8ed4-08e657dc7790" (UID: "34e13faa-3065-4742-8ed4-08e657dc7790"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:19:43.172547 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:43.172477    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34e13faa-3065-4742-8ed4-08e657dc7790-kube-api-access-65rkh" pod "34e13faa-3065-4742-8ed4-08e657dc7790" (UID: "34e13faa-3065-4742-8ed4-08e657dc7790"). InnerVolumeSpecName "kube-api-access-65rkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:19:43.271538 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:43.271502    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65rkh\" (UniqueName: \"kubernetes.io/projected/34e13faa-3065-4742-8ed4-08e657dc7790-kube-api-access-65rkh\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:19:43.271538 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:43.271531    4431 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34e13faa-3065-4742-8ed4-08e657dc7790-openshift-service-ca\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:19:43.271538 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:43.271544    4431 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34e13faa-3065-4742-8ed4-08e657dc7790-metrics-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:19:43.271766 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:43.271556    4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34e13faa-3065-4742-8ed4-08e657dc7790-maas-api-tls\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:19:43.849743 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:43.849707    4431 generic.go:356] "Generic (PLEG): container finished" podID="34e13faa-3065-4742-8ed4-08e657dc7790" containerID="5bc62380a4e4190acb8c75492e7e22c54fe1fe8abfe87207451e8fa7e97eef07" exitCode=0
Nov 03 22:19:43.850253 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:43.849766    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7dpdk"
Nov 03 22:19:43.850253 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:43.849801    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7dpdk" event={"ID":"34e13faa-3065-4742-8ed4-08e657dc7790","Type":"ContainerDied","Data":"5bc62380a4e4190acb8c75492e7e22c54fe1fe8abfe87207451e8fa7e97eef07"}
Nov 03 22:19:43.850253 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:43.849864    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7dpdk" event={"ID":"34e13faa-3065-4742-8ed4-08e657dc7790","Type":"ContainerDied","Data":"5da6893867ff74207ba4e6d216ad1fd13f13e3156690c54bfe5d84717ec0c036"}
Nov 03 22:19:43.850253 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:43.849903    4431 scope.go:122] "RemoveContainer" containerID="5bc62380a4e4190acb8c75492e7e22c54fe1fe8abfe87207451e8fa7e97eef07"
Nov 03 22:19:43.859559 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:43.859541    4431 scope.go:122] "RemoveContainer" containerID="5bc62380a4e4190acb8c75492e7e22c54fe1fe8abfe87207451e8fa7e97eef07"
Nov 03 22:19:43.859849 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:19:43.859831    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bc62380a4e4190acb8c75492e7e22c54fe1fe8abfe87207451e8fa7e97eef07\": container with ID starting with 5bc62380a4e4190acb8c75492e7e22c54fe1fe8abfe87207451e8fa7e97eef07 not found: ID does not exist" containerID="5bc62380a4e4190acb8c75492e7e22c54fe1fe8abfe87207451e8fa7e97eef07"
Nov 03 22:19:43.859904 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:43.859860    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bc62380a4e4190acb8c75492e7e22c54fe1fe8abfe87207451e8fa7e97eef07"} err="failed to get container status \"5bc62380a4e4190acb8c75492e7e22c54fe1fe8abfe87207451e8fa7e97eef07\": rpc error: code = NotFound desc = could not find container \"5bc62380a4e4190acb8c75492e7e22c54fe1fe8abfe87207451e8fa7e97eef07\": container with ID starting with 5bc62380a4e4190acb8c75492e7e22c54fe1fe8abfe87207451e8fa7e97eef07 not found: ID does not exist"
Nov 03 22:19:43.900382 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:43.900351    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-7dpdk"]
Nov 03 22:19:43.912758 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:43.912728    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-7dpdk"]
Nov 03 22:19:44.086953 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:44.086921    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34e13faa-3065-4742-8ed4-08e657dc7790" path="/var/lib/kubelet/pods/34e13faa-3065-4742-8ed4-08e657dc7790/volumes"
Nov 03 22:19:44.611458 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:44.611426    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:19:58.505115 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:58.505076    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-flhrd"]
Nov 03 22:19:58.505600 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:58.505574    4431 cpu_manager.go:414] "containerMap: removing container" podUID="34e13faa-3065-4742-8ed4-08e657dc7790" containerName="maas-api"
Nov 03 22:19:58.505600 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:58.505591    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34e13faa-3065-4742-8ed4-08e657dc7790" containerName="maas-api"
Nov 03 22:19:58.505672 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:58.505662    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="34e13faa-3065-4742-8ed4-08e657dc7790" containerName="maas-api"
Nov 03 22:19:58.509087 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:58.509071    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-flhrd"
Nov 03 22:19:58.528040 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:58.528013    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-flhrd"]
Nov 03 22:19:58.601532 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:58.601498    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ms9l\" (UniqueName: \"kubernetes.io/projected/21a175d3-b689-4352-91cb-dba3fe6f0247-kube-api-access-9ms9l\") pod \"llmisvc-controller-manager-6466c554d8-flhrd\" (UID: \"21a175d3-b689-4352-91cb-dba3fe6f0247\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-flhrd"
Nov 03 22:19:58.601683 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:58.601568    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/21a175d3-b689-4352-91cb-dba3fe6f0247-cert\") pod \"llmisvc-controller-manager-6466c554d8-flhrd\" (UID: \"21a175d3-b689-4352-91cb-dba3fe6f0247\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-flhrd"
Nov 03 22:19:58.702535 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:58.702500    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ms9l\" (UniqueName: \"kubernetes.io/projected/21a175d3-b689-4352-91cb-dba3fe6f0247-kube-api-access-9ms9l\") pod \"llmisvc-controller-manager-6466c554d8-flhrd\" (UID: \"21a175d3-b689-4352-91cb-dba3fe6f0247\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-flhrd"
Nov 03 22:19:58.702697 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:58.702566    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/21a175d3-b689-4352-91cb-dba3fe6f0247-cert\") pod \"llmisvc-controller-manager-6466c554d8-flhrd\" (UID: \"21a175d3-b689-4352-91cb-dba3fe6f0247\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-flhrd"
Nov 03 22:19:58.704975 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:58.704948    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/21a175d3-b689-4352-91cb-dba3fe6f0247-cert\") pod \"llmisvc-controller-manager-6466c554d8-flhrd\" (UID: \"21a175d3-b689-4352-91cb-dba3fe6f0247\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-flhrd"
Nov 03 22:19:58.725780 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:58.725756    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ms9l\" (UniqueName: \"kubernetes.io/projected/21a175d3-b689-4352-91cb-dba3fe6f0247-kube-api-access-9ms9l\") pod \"llmisvc-controller-manager-6466c554d8-flhrd\" (UID: \"21a175d3-b689-4352-91cb-dba3fe6f0247\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-flhrd"
Nov 03 22:19:58.821717 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:58.821632    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-flhrd"
Nov 03 22:19:59.114702 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:59.114676    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-flhrd"]
Nov 03 22:19:59.116853 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:19:59.116824    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21a175d3_b689_4352_91cb_dba3fe6f0247.slice/crio-fd6726137b4b1dbb0be056b1f22c118b9f42ab4f6cca69058328e24eaad54d28 WatchSource:0}: Error finding container fd6726137b4b1dbb0be056b1f22c118b9f42ab4f6cca69058328e24eaad54d28: Status 404 returned error can't find the container with id fd6726137b4b1dbb0be056b1f22c118b9f42ab4f6cca69058328e24eaad54d28
Nov 03 22:19:59.919328 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:59.919279    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-flhrd" event={"ID":"21a175d3-b689-4352-91cb-dba3fe6f0247","Type":"ContainerStarted","Data":"bbe3d1988a061337988bf2d247f9148e10bfaa1cd3b2bbb2018dce18c95c92eb"}
Nov 03 22:19:59.919723 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:59.919432    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-flhrd"
Nov 03 22:19:59.919723 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:59.919460    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-flhrd" event={"ID":"21a175d3-b689-4352-91cb-dba3fe6f0247","Type":"ContainerStarted","Data":"fd6726137b4b1dbb0be056b1f22c118b9f42ab4f6cca69058328e24eaad54d28"}
Nov 03 22:19:59.961534 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:19:59.961470    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-flhrd" podStartSLOduration=1.562290884 podStartE2EDuration="1.961449186s" totalImagesPullingTime="399.158302ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:19:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:19:59.95952925 +0000 UTC m=+423.985385112" watchObservedRunningTime="2026-11-03 22:19:59.961449186 +0000 UTC m=+423.987305049"
Nov 03 22:20:01.258034 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:20:01.257998    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:20:30.926528 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:20:30.926496    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-flhrd"
Nov 03 22:20:31.007344 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:20:31.007313    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-4zggb"]
Nov 03 22:20:31.007552 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:20:31.007517    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4zggb" podUID="c811b17a-e08a-43e2-8986-59991e1a9804" containerName="manager" containerID="cri-o://de1c73759fc74bfb1161059f1a522fba019508e9d93cbe6ec18556d9aff2cef6" gracePeriod=30
Nov 03 22:20:31.254800 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:20:31.254774    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4zggb"
Nov 03 22:20:31.258135 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:20:31.258103    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:20:31.282911 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:20:31.282887    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c811b17a-e08a-43e2-8986-59991e1a9804-cert\" (UniqueName: \"kubernetes.io/secret/c811b17a-e08a-43e2-8986-59991e1a9804-cert\") pod \"c811b17a-e08a-43e2-8986-59991e1a9804\" (UID: \"c811b17a-e08a-43e2-8986-59991e1a9804\") "
Nov 03 22:20:31.283044 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:20:31.282975    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c811b17a-e08a-43e2-8986-59991e1a9804-kube-api-access-n2hrf\" (UniqueName: \"kubernetes.io/projected/c811b17a-e08a-43e2-8986-59991e1a9804-kube-api-access-n2hrf\") pod \"c811b17a-e08a-43e2-8986-59991e1a9804\" (UID: \"c811b17a-e08a-43e2-8986-59991e1a9804\") "
Nov 03 22:20:31.285013 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:20:31.284990    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c811b17a-e08a-43e2-8986-59991e1a9804-cert" pod "c811b17a-e08a-43e2-8986-59991e1a9804" (UID: "c811b17a-e08a-43e2-8986-59991e1a9804"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:20:31.285117 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:20:31.285061    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c811b17a-e08a-43e2-8986-59991e1a9804-kube-api-access-n2hrf" pod "c811b17a-e08a-43e2-8986-59991e1a9804" (UID: "c811b17a-e08a-43e2-8986-59991e1a9804"). InnerVolumeSpecName "kube-api-access-n2hrf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:20:31.384175 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:20:31.384138    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2hrf\" (UniqueName: \"kubernetes.io/projected/c811b17a-e08a-43e2-8986-59991e1a9804-kube-api-access-n2hrf\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:20:31.384175 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:20:31.384169    4431 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c811b17a-e08a-43e2-8986-59991e1a9804-cert\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:20:31.871409 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:20:31.871370    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:20:32.012917 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:20:32.012885    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:20:32.054849 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:20:32.054814    4431 generic.go:356] "Generic (PLEG): container finished" podID="c811b17a-e08a-43e2-8986-59991e1a9804" containerID="de1c73759fc74bfb1161059f1a522fba019508e9d93cbe6ec18556d9aff2cef6" exitCode=0
Nov 03 22:20:32.055049 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:20:32.054868    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4zggb"
Nov 03 22:20:32.055049 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:20:32.054905    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4zggb" event={"ID":"c811b17a-e08a-43e2-8986-59991e1a9804","Type":"ContainerDied","Data":"de1c73759fc74bfb1161059f1a522fba019508e9d93cbe6ec18556d9aff2cef6"}
Nov 03 22:20:32.055049 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:20:32.054970    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4zggb" event={"ID":"c811b17a-e08a-43e2-8986-59991e1a9804","Type":"ContainerDied","Data":"be74d2fe6f591ada7d6b732a2cac54c68cdd404e5c918f90f3aa9bdc24000ebe"}
Nov 03 22:20:32.055049 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:20:32.055000    4431 scope.go:122] "RemoveContainer" containerID="de1c73759fc74bfb1161059f1a522fba019508e9d93cbe6ec18556d9aff2cef6"
Nov 03 22:20:32.063792 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:20:32.063774    4431 scope.go:122] "RemoveContainer" containerID="de1c73759fc74bfb1161059f1a522fba019508e9d93cbe6ec18556d9aff2cef6"
Nov 03 22:20:32.064066 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:20:32.064046    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de1c73759fc74bfb1161059f1a522fba019508e9d93cbe6ec18556d9aff2cef6\": container with ID starting with de1c73759fc74bfb1161059f1a522fba019508e9d93cbe6ec18556d9aff2cef6 not found: ID does not exist" containerID="de1c73759fc74bfb1161059f1a522fba019508e9d93cbe6ec18556d9aff2cef6"
Nov 03 22:20:32.064133 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:20:32.064079    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de1c73759fc74bfb1161059f1a522fba019508e9d93cbe6ec18556d9aff2cef6"} err="failed to get container status \"de1c73759fc74bfb1161059f1a522fba019508e9d93cbe6ec18556d9aff2cef6\": rpc error: code = NotFound desc = could not find container \"de1c73759fc74bfb1161059f1a522fba019508e9d93cbe6ec18556d9aff2cef6\": container with ID starting with de1c73759fc74bfb1161059f1a522fba019508e9d93cbe6ec18556d9aff2cef6 not found: ID does not exist"
Nov 03 22:20:32.105292 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:20:32.105257    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-4zggb"]
Nov 03 22:20:32.117596 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:20:32.117570    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-4zggb"]
Nov 03 22:20:33.237446 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:20:33.237411    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:20:34.085163 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:20:34.085132    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c811b17a-e08a-43e2-8986-59991e1a9804" path="/var/lib/kubelet/pods/c811b17a-e08a-43e2-8986-59991e1a9804/volumes"
Nov 03 22:20:44.733001 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:20:44.732968    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:20:48.363766 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:20:48.363733    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:20:48.503068 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:20:48.503037    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:20:49.440559 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:20:49.440523    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:20:51.505865 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:20:51.505828    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:20:52.881697 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:20:52.881651    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:20:53.273170 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:20:53.273140    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:20:53.412653 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:20:53.412619    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:20:54.177946 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:20:54.177911    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:20:55.814105 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:20:55.814070    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:20:57.984457 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:20:57.984422    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:21:00.619153 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:00.619117    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:21:01.258368 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:21:01.258332    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:21:01.992864 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:01.992801    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:21:13.240294 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:13.240198    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-d6hr4"]
Nov 03 22:21:13.240789 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:13.240768    4431 cpu_manager.go:414] "containerMap: removing container" podUID="c811b17a-e08a-43e2-8986-59991e1a9804" containerName="manager"
Nov 03 22:21:13.240861 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:13.240792    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c811b17a-e08a-43e2-8986-59991e1a9804" containerName="manager"
Nov 03 22:21:13.240922 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:13.240911    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="c811b17a-e08a-43e2-8986-59991e1a9804" containerName="manager"
Nov 03 22:21:13.243934 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:13.243912    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-d6hr4"
Nov 03 22:21:13.276950 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:13.276909    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-d6hr4"]
Nov 03 22:21:13.344684 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:13.344653    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 03 22:21:13.344837 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:13.344733    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-9d7jc\""
Nov 03 22:21:13.352145 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:13.352116    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9286b874-939f-479a-b771-a4e877cbd50f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-d6hr4\" (UID: \"9286b874-939f-479a-b771-a4e877cbd50f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-d6hr4"
Nov 03 22:21:13.352309 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:13.352161    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9286b874-939f-479a-b771-a4e877cbd50f-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-d6hr4\" (UID: \"9286b874-939f-479a-b771-a4e877cbd50f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-d6hr4"
Nov 03 22:21:13.352309 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:13.352178    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9286b874-939f-479a-b771-a4e877cbd50f-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-d6hr4\" (UID: \"9286b874-939f-479a-b771-a4e877cbd50f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-d6hr4"
Nov 03 22:21:13.352389 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:13.352306    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9286b874-939f-479a-b771-a4e877cbd50f-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-d6hr4\" (UID: \"9286b874-939f-479a-b771-a4e877cbd50f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-d6hr4"
Nov 03 22:21:13.352389 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:13.352327    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9286b874-939f-479a-b771-a4e877cbd50f-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-d6hr4\" (UID: \"9286b874-939f-479a-b771-a4e877cbd50f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-d6hr4"
Nov 03 22:21:13.352389 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:13.352355    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9286b874-939f-479a-b771-a4e877cbd50f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-d6hr4\" (UID: \"9286b874-939f-479a-b771-a4e877cbd50f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-d6hr4"
Nov 03 22:21:13.352389 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:13.352374    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9286b874-939f-479a-b771-a4e877cbd50f-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-d6hr4\" (UID: \"9286b874-939f-479a-b771-a4e877cbd50f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-d6hr4"
Nov 03 22:21:13.352510 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:13.352414    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hf9ft\" (UniqueName: \"kubernetes.io/projected/9286b874-939f-479a-b771-a4e877cbd50f-kube-api-access-hf9ft\") pod \"maas-default-gateway-openshift-default-7df854bf67-d6hr4\" (UID: \"9286b874-939f-479a-b771-a4e877cbd50f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-d6hr4"
Nov 03 22:21:13.352546 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:13.352519    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9286b874-939f-479a-b771-a4e877cbd50f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-d6hr4\" (UID: \"9286b874-939f-479a-b771-a4e877cbd50f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-d6hr4"
Nov 03 22:21:13.453379 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:13.453343    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9286b874-939f-479a-b771-a4e877cbd50f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-d6hr4\" (UID: \"9286b874-939f-479a-b771-a4e877cbd50f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-d6hr4"
Nov 03 22:21:13.453547 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:13.453387    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9286b874-939f-479a-b771-a4e877cbd50f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-d6hr4\" (UID: \"9286b874-939f-479a-b771-a4e877cbd50f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-d6hr4"
Nov 03 22:21:13.453547 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:13.453413    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9286b874-939f-479a-b771-a4e877cbd50f-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-d6hr4\" (UID: \"9286b874-939f-479a-b771-a4e877cbd50f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-d6hr4"
Nov 03 22:21:13.453547 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:13.453428    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9286b874-939f-479a-b771-a4e877cbd50f-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-d6hr4\" (UID: \"9286b874-939f-479a-b771-a4e877cbd50f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-d6hr4"
Nov 03 22:21:13.453689 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:13.453549    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9286b874-939f-479a-b771-a4e877cbd50f-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-d6hr4\" (UID: \"9286b874-939f-479a-b771-a4e877cbd50f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-d6hr4"
Nov 03 22:21:13.453689 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:13.453579    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9286b874-939f-479a-b771-a4e877cbd50f-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-d6hr4\" (UID: \"9286b874-939f-479a-b771-a4e877cbd50f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-d6hr4"
Nov 03 22:21:13.453689 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:13.453622    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9286b874-939f-479a-b771-a4e877cbd50f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-d6hr4\" (UID: \"9286b874-939f-479a-b771-a4e877cbd50f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-d6hr4"
Nov 03 22:21:13.453689 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:13.453649    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9286b874-939f-479a-b771-a4e877cbd50f-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-d6hr4\" (UID: \"9286b874-939f-479a-b771-a4e877cbd50f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-d6hr4"
Nov 03 22:21:13.453689 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:13.453678    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hf9ft\" (UniqueName: \"kubernetes.io/projected/9286b874-939f-479a-b771-a4e877cbd50f-kube-api-access-hf9ft\") pod \"maas-default-gateway-openshift-default-7df854bf67-d6hr4\" (UID: \"9286b874-939f-479a-b771-a4e877cbd50f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-d6hr4"
Nov 03 22:21:13.453941 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:13.453856    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9286b874-939f-479a-b771-a4e877cbd50f-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-d6hr4\" (UID: \"9286b874-939f-479a-b771-a4e877cbd50f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-d6hr4"
Nov 03 22:21:13.454038 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:13.454014    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9286b874-939f-479a-b771-a4e877cbd50f-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-d6hr4\" (UID: \"9286b874-939f-479a-b771-a4e877cbd50f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-d6hr4"
Nov 03 22:21:13.454112 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:13.454065    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9286b874-939f-479a-b771-a4e877cbd50f-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-d6hr4\" (UID: \"9286b874-939f-479a-b771-a4e877cbd50f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-d6hr4"
Nov 03 22:21:13.454176 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:13.454117    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9286b874-939f-479a-b771-a4e877cbd50f-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-d6hr4\" (UID: \"9286b874-939f-479a-b771-a4e877cbd50f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-d6hr4"
Nov 03 22:21:13.454303 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:13.454169    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9286b874-939f-479a-b771-a4e877cbd50f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-d6hr4\" (UID: \"9286b874-939f-479a-b771-a4e877cbd50f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-d6hr4"
Nov 03 22:21:13.455801 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:13.455779    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9286b874-939f-479a-b771-a4e877cbd50f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-d6hr4\" (UID: \"9286b874-939f-479a-b771-a4e877cbd50f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-d6hr4"
Nov 03 22:21:13.456019 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:13.456003    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9286b874-939f-479a-b771-a4e877cbd50f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-d6hr4\" (UID: \"9286b874-939f-479a-b771-a4e877cbd50f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-d6hr4"
Nov 03 22:21:13.476864 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:13.476841    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9286b874-939f-479a-b771-a4e877cbd50f-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-d6hr4\" (UID: \"9286b874-939f-479a-b771-a4e877cbd50f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-d6hr4"
Nov 03 22:21:13.477027 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:13.477010    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hf9ft\" (UniqueName: \"kubernetes.io/projected/9286b874-939f-479a-b771-a4e877cbd50f-kube-api-access-hf9ft\") pod \"maas-default-gateway-openshift-default-7df854bf67-d6hr4\" (UID: \"9286b874-939f-479a-b771-a4e877cbd50f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-d6hr4"
Nov 03 22:21:13.557390 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:13.557308    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-d6hr4"
Nov 03 22:21:13.871065 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:13.871040    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-d6hr4"]
Nov 03 22:21:13.873191 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:21:13.873165    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9286b874_939f_479a_b771_a4e877cbd50f.slice/crio-bb802e136c290b589cbfa360117486ef138c0c8275e7bd692bf303562b0702ab WatchSource:0}: Error finding container bb802e136c290b589cbfa360117486ef138c0c8275e7bd692bf303562b0702ab: Status 404 returned error can't find the container with id bb802e136c290b589cbfa360117486ef138c0c8275e7bd692bf303562b0702ab
Nov 03 22:21:14.241094 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:14.241057    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-d6hr4" event={"ID":"9286b874-939f-479a-b771-a4e877cbd50f","Type":"ContainerStarted","Data":"bb802e136c290b589cbfa360117486ef138c0c8275e7bd692bf303562b0702ab"}
Nov 03 22:21:16.560459 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:16.560412    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:21:16.668340 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:16.668310    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:21:16.668435 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:16.668375    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:21:16.668435 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:16.668403    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:21:16.705301 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:16.705274    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:21:17.266130 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:17.266090    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-d6hr4" event={"ID":"9286b874-939f-479a-b771-a4e877cbd50f","Type":"ContainerStarted","Data":"d2a10f47e05277e53af0a2e01ed45f476a16b28cfbe7b7cc8098a7a81e9362da"}
Nov 03 22:21:17.328056 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:17.327997    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-d6hr4" podStartSLOduration=1.535899899 podStartE2EDuration="4.327984111s" totalImagesPullingTime="2.792084212s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:21:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:21:17.321760454 +0000 UTC m=+501.347616317" watchObservedRunningTime="2026-11-03 22:21:17.327984111 +0000 UTC m=+501.353839972"
Nov 03 22:21:17.558679 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:17.558550    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-d6hr4"
Nov 03 22:21:17.563134 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:17.562769    4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-d6hr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Nov 03 22:21:17.563134 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:17.562837    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-d6hr4" podUID="9286b874-939f-479a-b771-a4e877cbd50f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Nov 03 22:21:18.558704 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:18.558666    4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-d6hr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Nov 03 22:21:18.558981 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:18.558952    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-d6hr4" podUID="9286b874-939f-479a-b771-a4e877cbd50f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Nov 03 22:21:19.558692 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:19.558651    4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-d6hr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Nov 03 22:21:19.559313 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:19.559272    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-d6hr4" podUID="9286b874-939f-479a-b771-a4e877cbd50f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Nov 03 22:21:20.560665 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:20.560630    4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-d6hr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Nov 03 22:21:20.561273 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:20.561241    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-d6hr4" podUID="9286b874-939f-479a-b771-a4e877cbd50f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Nov 03 22:21:21.558648 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:21.558608    4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-d6hr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Nov 03 22:21:21.558958 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:21.558927    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-d6hr4" podUID="9286b874-939f-479a-b771-a4e877cbd50f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Nov 03 22:21:22.558462 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:22.558425    4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-d6hr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Nov 03 22:21:22.558865 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:22.558486    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-d6hr4" podUID="9286b874-939f-479a-b771-a4e877cbd50f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Nov 03 22:21:22.634971 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:22.634940    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:21:22.807951 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:22.807907    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:21:23.558700 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:23.558655    4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-d6hr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Nov 03 22:21:23.559105 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:23.558745    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-d6hr4" podUID="9286b874-939f-479a-b771-a4e877cbd50f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Nov 03 22:21:25.558937 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:25.558898    4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-d6hr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 22:21:25.559382 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:25.558950    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-d6hr4" podUID="9286b874-939f-479a-b771-a4e877cbd50f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:21:26.600417 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:26.600372    4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-d6hr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 22:21:26.600904 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:26.600444    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-d6hr4" podUID="9286b874-939f-479a-b771-a4e877cbd50f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:21:26.949759 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:26.949725    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-d6hr4"
Nov 03 22:21:26.950056 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:26.950017    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-d6hr4"
Nov 03 22:21:26.952031 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:26.952004    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-d6hr4"
Nov 03 22:21:28.284387 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:28.284349    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-r4qjb"]
Nov 03 22:21:28.288404 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:28.288382    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-r4qjb"
Nov 03 22:21:28.321426 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:28.321395    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-r4qjb"]
Nov 03 22:21:28.375913 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:28.375879    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8xff\" (UniqueName: \"kubernetes.io/projected/d5687c54-7c44-4806-846e-3646f096b7ab-kube-api-access-c8xff\") pod \"maas-default-gateway-openshift-default-7df854bf67-r4qjb\" (UID: \"d5687c54-7c44-4806-846e-3646f096b7ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-r4qjb"
Nov 03 22:21:28.376109 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:28.375929    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5687c54-7c44-4806-846e-3646f096b7ab-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-r4qjb\" (UID: \"d5687c54-7c44-4806-846e-3646f096b7ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-r4qjb"
Nov 03 22:21:28.376109 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:28.376022    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5687c54-7c44-4806-846e-3646f096b7ab-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-r4qjb\" (UID: \"d5687c54-7c44-4806-846e-3646f096b7ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-r4qjb"
Nov 03 22:21:28.376109 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:28.376041    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5687c54-7c44-4806-846e-3646f096b7ab-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-r4qjb\" (UID: \"d5687c54-7c44-4806-846e-3646f096b7ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-r4qjb"
Nov 03 22:21:28.376109 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:28.376102    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5687c54-7c44-4806-846e-3646f096b7ab-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-r4qjb\" (UID: \"d5687c54-7c44-4806-846e-3646f096b7ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-r4qjb"
Nov 03 22:21:28.376374 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:28.376171    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5687c54-7c44-4806-846e-3646f096b7ab-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-r4qjb\" (UID: \"d5687c54-7c44-4806-846e-3646f096b7ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-r4qjb"
Nov 03 22:21:28.376374 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:28.376187    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5687c54-7c44-4806-846e-3646f096b7ab-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-r4qjb\" (UID: \"d5687c54-7c44-4806-846e-3646f096b7ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-r4qjb"
Nov 03 22:21:28.376374 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:28.376242    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5687c54-7c44-4806-846e-3646f096b7ab-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-r4qjb\" (UID: \"d5687c54-7c44-4806-846e-3646f096b7ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-r4qjb"
Nov 03 22:21:28.376374 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:28.376362    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5687c54-7c44-4806-846e-3646f096b7ab-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-r4qjb\" (UID: \"d5687c54-7c44-4806-846e-3646f096b7ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-r4qjb"
Nov 03 22:21:28.477508 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:28.477472    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5687c54-7c44-4806-846e-3646f096b7ab-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-r4qjb\" (UID: \"d5687c54-7c44-4806-846e-3646f096b7ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-r4qjb"
Nov 03 22:21:28.477663 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:28.477518    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8xff\" (UniqueName: \"kubernetes.io/projected/d5687c54-7c44-4806-846e-3646f096b7ab-kube-api-access-c8xff\") pod \"maas-default-gateway-openshift-default-7df854bf67-r4qjb\" (UID: \"d5687c54-7c44-4806-846e-3646f096b7ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-r4qjb"
Nov 03 22:21:28.477663 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:28.477551    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5687c54-7c44-4806-846e-3646f096b7ab-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-r4qjb\" (UID: \"d5687c54-7c44-4806-846e-3646f096b7ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-r4qjb"
Nov 03 22:21:28.477663 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:28.477585    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5687c54-7c44-4806-846e-3646f096b7ab-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-r4qjb\" (UID: \"d5687c54-7c44-4806-846e-3646f096b7ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-r4qjb"
Nov 03 22:21:28.477663 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:28.477608    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5687c54-7c44-4806-846e-3646f096b7ab-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-r4qjb\" (UID: \"d5687c54-7c44-4806-846e-3646f096b7ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-r4qjb"
Nov 03 22:21:28.477663 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:28.477640    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5687c54-7c44-4806-846e-3646f096b7ab-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-r4qjb\" (UID: \"d5687c54-7c44-4806-846e-3646f096b7ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-r4qjb"
Nov 03 22:21:28.477930 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:28.477697    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5687c54-7c44-4806-846e-3646f096b7ab-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-r4qjb\" (UID: \"d5687c54-7c44-4806-846e-3646f096b7ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-r4qjb"
Nov 03 22:21:28.477930 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:28.477725    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5687c54-7c44-4806-846e-3646f096b7ab-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-r4qjb\" (UID: \"d5687c54-7c44-4806-846e-3646f096b7ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-r4qjb"
Nov 03 22:21:28.477930 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:28.477757    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5687c54-7c44-4806-846e-3646f096b7ab-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-r4qjb\" (UID: \"d5687c54-7c44-4806-846e-3646f096b7ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-r4qjb"
Nov 03 22:21:28.478084 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:28.477929    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5687c54-7c44-4806-846e-3646f096b7ab-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-r4qjb\" (UID: \"d5687c54-7c44-4806-846e-3646f096b7ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-r4qjb"
Nov 03 22:21:28.478084 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:28.477958    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5687c54-7c44-4806-846e-3646f096b7ab-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-r4qjb\" (UID: \"d5687c54-7c44-4806-846e-3646f096b7ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-r4qjb"
Nov 03 22:21:28.478084 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:28.478036    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5687c54-7c44-4806-846e-3646f096b7ab-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-r4qjb\" (UID: \"d5687c54-7c44-4806-846e-3646f096b7ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-r4qjb"
Nov 03 22:21:28.478259 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:28.478110    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5687c54-7c44-4806-846e-3646f096b7ab-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-r4qjb\" (UID: \"d5687c54-7c44-4806-846e-3646f096b7ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-r4qjb"
Nov 03 22:21:28.478449 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:28.478429    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5687c54-7c44-4806-846e-3646f096b7ab-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-r4qjb\" (UID: \"d5687c54-7c44-4806-846e-3646f096b7ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-r4qjb"
Nov 03 22:21:28.480145 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:28.480113    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5687c54-7c44-4806-846e-3646f096b7ab-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-r4qjb\" (UID: \"d5687c54-7c44-4806-846e-3646f096b7ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-r4qjb"
Nov 03 22:21:28.480312 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:28.480296    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5687c54-7c44-4806-846e-3646f096b7ab-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-r4qjb\" (UID: \"d5687c54-7c44-4806-846e-3646f096b7ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-r4qjb"
Nov 03 22:21:28.501191 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:28.501161    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5687c54-7c44-4806-846e-3646f096b7ab-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-r4qjb\" (UID: \"d5687c54-7c44-4806-846e-3646f096b7ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-r4qjb"
Nov 03 22:21:28.501341 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:28.501322    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8xff\" (UniqueName: \"kubernetes.io/projected/d5687c54-7c44-4806-846e-3646f096b7ab-kube-api-access-c8xff\") pod \"maas-default-gateway-openshift-default-7df854bf67-r4qjb\" (UID: \"d5687c54-7c44-4806-846e-3646f096b7ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-r4qjb"
Nov 03 22:21:28.603508 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:28.603433    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-r4qjb"
Nov 03 22:21:28.917395 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:28.917328    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-r4qjb"]
Nov 03 22:21:28.920116 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:21:28.920090    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5687c54_7c44_4806_846e_3646f096b7ab.slice/crio-6a92df69fa358206bac87ae467cea04492ccc5315b0a1e398d8104d3abc36037 WatchSource:0}: Error finding container 6a92df69fa358206bac87ae467cea04492ccc5315b0a1e398d8104d3abc36037: Status 404 returned error can't find the container with id 6a92df69fa358206bac87ae467cea04492ccc5315b0a1e398d8104d3abc36037
Nov 03 22:21:28.922341 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:28.922299    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:21:28.922465 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:28.922384    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:21:28.922465 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:28.922428    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:21:29.324834 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:29.324749    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-r4qjb" event={"ID":"d5687c54-7c44-4806-846e-3646f096b7ab","Type":"ContainerStarted","Data":"82e28828aef5fc4f494a32d2b0c0df6bbe6b63265cb1902de1b739d7e45ba38d"}
Nov 03 22:21:29.324834 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:29.324801    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-r4qjb" event={"ID":"d5687c54-7c44-4806-846e-3646f096b7ab","Type":"ContainerStarted","Data":"6a92df69fa358206bac87ae467cea04492ccc5315b0a1e398d8104d3abc36037"}
Nov 03 22:21:29.394445 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:29.394394    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-r4qjb" podStartSLOduration=1.394380474 podStartE2EDuration="1.394380474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:21:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:21:29.382094759 +0000 UTC m=+513.407950632" watchObservedRunningTime="2026-11-03 22:21:29.394380474 +0000 UTC m=+513.420236336"
Nov 03 22:21:29.603899 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:29.603806    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-r4qjb"
Nov 03 22:21:29.605395 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:29.605369    4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-r4qjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Nov 03 22:21:29.605516 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:29.605417    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-r4qjb" podUID="d5687c54-7c44-4806-846e-3646f096b7ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Nov 03 22:21:30.603996 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:30.603963    4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-r4qjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Nov 03 22:21:30.604421 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:30.604016    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-r4qjb" podUID="d5687c54-7c44-4806-846e-3646f096b7ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Nov 03 22:21:31.282790 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:21:31.282752    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:21:31.604636 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:31.604552    4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-r4qjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Nov 03 22:21:31.604636 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:31.604606    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-r4qjb" podUID="d5687c54-7c44-4806-846e-3646f096b7ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Nov 03 22:21:32.603919 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:32.603887    4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-r4qjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Nov 03 22:21:32.604101 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:32.603943    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-r4qjb" podUID="d5687c54-7c44-4806-846e-3646f096b7ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Nov 03 22:21:33.604656 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:33.604618    4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-r4qjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Nov 03 22:21:33.605250 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:33.604677    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-r4qjb" podUID="d5687c54-7c44-4806-846e-3646f096b7ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Nov 03 22:21:34.604799 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:34.604759    4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-r4qjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Nov 03 22:21:34.605243 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:34.604823    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-r4qjb" podUID="d5687c54-7c44-4806-846e-3646f096b7ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Nov 03 22:21:35.604055 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:35.604014    4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-r4qjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Nov 03 22:21:35.604259 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:35.604091    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-r4qjb" podUID="d5687c54-7c44-4806-846e-3646f096b7ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Nov 03 22:21:37.162627 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:37.162590    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-r4qjb"
Nov 03 22:21:37.163086 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:37.163001    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-r4qjb"
Nov 03 22:21:37.163837 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:37.163816    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-r4qjb"
Nov 03 22:21:44.851257 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:44.851198    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:21:46.416098 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:46.416053    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:21:46.915671 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:46.915634    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:21:48.477987 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:48.477951    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:21:48.617168 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:48.617101    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:21:50.051360 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:50.051329    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:21:50.194907 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:21:50.194871    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:01.257287 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:22:01.257245    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:22:07.155564 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:07.155528    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:11.661066 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:11.661017    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:11.803371 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:11.803334    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:12.141298 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:12.141263    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:12.305889 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:12.305862    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:12.463301 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:12.463266    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:12.610773 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:12.610742    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:12.760284 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:12.760185    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:12.779029 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:12.778990    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96"]
Nov 03 22:22:12.783525 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:12.783499    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96"
Nov 03 22:22:12.802919 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:12.802889    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-e19850-openshift-default-dockercfg-5fvb8\""
Nov 03 22:22:12.817971 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:12.817938    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96"]
Nov 03 22:22:12.877956 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:12.877921    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c-istio-podinfo\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96\" (UID: \"b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96"
Nov 03 22:22:12.877956 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:12.877964    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwhgp\" (UniqueName: \"kubernetes.io/projected/b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c-kube-api-access-fwhgp\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96\" (UID: \"b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96"
Nov 03 22:22:12.878183 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:12.878001    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c-istio-data\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96\" (UID: \"b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96"
Nov 03 22:22:12.878183 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:12.878148    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c-credential-socket\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96\" (UID: \"b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96"
Nov 03 22:22:12.878283 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:12.878198    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c-workload-socket\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96\" (UID: \"b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96"
Nov 03 22:22:12.878366 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:12.878340    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c-workload-certs\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96\" (UID: \"b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96"
Nov 03 22:22:12.878416 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:12.878380    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c-istio-token\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96\" (UID: \"b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96"
Nov 03 22:22:12.878416 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:12.878406    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c-istiod-ca-cert\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96\" (UID: \"b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96"
Nov 03 22:22:12.878560 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:12.878530    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c-istio-envoy\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96\" (UID: \"b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96"
Nov 03 22:22:12.909871 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:12.909831    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:12.979125 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:12.979083    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c-istio-envoy\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96\" (UID: \"b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96"
Nov 03 22:22:12.979325 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:12.979142    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c-istio-podinfo\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96\" (UID: \"b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96"
Nov 03 22:22:12.979325 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:12.979169    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwhgp\" (UniqueName: \"kubernetes.io/projected/b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c-kube-api-access-fwhgp\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96\" (UID: \"b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96"
Nov 03 22:22:12.979325 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:12.979195    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c-istio-data\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96\" (UID: \"b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96"
Nov 03 22:22:12.979325 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:12.979258    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c-credential-socket\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96\" (UID: \"b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96"
Nov 03 22:22:12.979325 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:12.979291    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c-workload-socket\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96\" (UID: \"b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96"
Nov 03 22:22:12.979606 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:12.979330    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c-workload-certs\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96\" (UID: \"b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96"
Nov 03 22:22:12.979606 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:12.979354    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c-istio-token\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96\" (UID: \"b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96"
Nov 03 22:22:12.979606 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:12.979380    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c-istiod-ca-cert\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96\" (UID: \"b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96"
Nov 03 22:22:12.979768 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:12.979677    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c-credential-socket\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96\" (UID: \"b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96"
Nov 03 22:22:12.979768 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:12.979716    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c-istio-data\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96\" (UID: \"b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96"
Nov 03 22:22:12.979889 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:12.979866    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c-workload-socket\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96\" (UID: \"b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96"
Nov 03 22:22:12.980078 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:12.980056    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c-workload-certs\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96\" (UID: \"b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96"
Nov 03 22:22:12.980169 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:12.980158    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c-istiod-ca-cert\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96\" (UID: \"b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96"
Nov 03 22:22:12.981436 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:12.981416    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c-istio-envoy\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96\" (UID: \"b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96"
Nov 03 22:22:12.982275 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:12.982255    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c-istio-podinfo\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96\" (UID: \"b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96"
Nov 03 22:22:13.002505 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:13.002470    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c-istio-token\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96\" (UID: \"b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96"
Nov 03 22:22:13.002629 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:13.002600    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwhgp\" (UniqueName: \"kubernetes.io/projected/b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c-kube-api-access-fwhgp\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96\" (UID: \"b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96"
Nov 03 22:22:13.059779 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:13.059672    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:13.101759 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:13.101716    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96"
Nov 03 22:22:13.271569 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:13.271535    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:13.418656 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:13.418335    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96"]
Nov 03 22:22:13.421303 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:22:13.421269    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb960a1f1_6b5e_4599_8e2e_3d4a13dc8b6c.slice/crio-98689f08b499e711c88d128c6b0592992e3236068aee01afde16d27d51caff34 WatchSource:0}: Error finding container 98689f08b499e711c88d128c6b0592992e3236068aee01afde16d27d51caff34: Status 404 returned error can't find the container with id 98689f08b499e711c88d128c6b0592992e3236068aee01afde16d27d51caff34
Nov 03 22:22:13.423589 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:13.423549    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:22:13.423659 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:13.423640    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:22:13.423696 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:13.423680    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:22:13.539478 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:13.539441    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96" event={"ID":"b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c","Type":"ContainerStarted","Data":"be9f70a522fa1e9bb79e5fde7dfc3cce48574a605cdbf375d5d5807d2e025250"}
Nov 03 22:22:13.539673 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:13.539504    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96" event={"ID":"b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c","Type":"ContainerStarted","Data":"98689f08b499e711c88d128c6b0592992e3236068aee01afde16d27d51caff34"}
Nov 03 22:22:13.586063 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:13.586012    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96" podStartSLOduration=1.5859984200000001 podStartE2EDuration="1.58599842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:22:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:13.584143064 +0000 UTC m=+557.609998927" watchObservedRunningTime="2026-11-03 22:22:13.58599842 +0000 UTC m=+557.611854281"
Nov 03 22:22:14.102899 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:14.102864    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96"
Nov 03 22:22:14.108129 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:14.108104    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96"
Nov 03 22:22:14.544683 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:14.544653    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96"
Nov 03 22:22:14.545740 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:14.545718    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96"
Nov 03 22:22:15.662067 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:15.662032    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:15.815940 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:15.815902    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:15.968027 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:15.967944    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:16.122596 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:16.122554    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:16.270271 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:16.270164    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:16.498662 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:16.498630    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:16.642547 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:16.642457    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:16.793891 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:16.793838    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:17.007560 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:17.007521    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:17.320388 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:17.320310    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:17.472094 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:17.472045    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:18.659852 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:18.659813    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-2d619f-6979556cf5-4sd4b"]
Nov 03 22:22:18.663800 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:18.663774    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-2d619f-6979556cf5-4sd4b"
Nov 03 22:22:18.688797 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:18.688765    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-2d619f-serving-cert\""
Nov 03 22:22:18.692296 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:18.692265    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-2d619f-6979556cf5-4sd4b"]
Nov 03 22:22:18.735607 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:18.735576    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab4a1779-440d-442f-bc77-751d89b544c7-maas-api-tls\") pod \"maas-api-e2e-worker-w6-2d619f-6979556cf5-4sd4b\" (UID: \"ab4a1779-440d-442f-bc77-751d89b544c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-2d619f-6979556cf5-4sd4b"
Nov 03 22:22:18.735764 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:18.735637    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab4a1779-440d-442f-bc77-751d89b544c7-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-2d619f-6979556cf5-4sd4b\" (UID: \"ab4a1779-440d-442f-bc77-751d89b544c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-2d619f-6979556cf5-4sd4b"
Nov 03 22:22:18.735849 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:18.735825    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh2rz\" (UniqueName: \"kubernetes.io/projected/ab4a1779-440d-442f-bc77-751d89b544c7-kube-api-access-lh2rz\") pod \"maas-api-e2e-worker-w6-2d619f-6979556cf5-4sd4b\" (UID: \"ab4a1779-440d-442f-bc77-751d89b544c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-2d619f-6979556cf5-4sd4b"
Nov 03 22:22:18.735893 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:18.735877    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab4a1779-440d-442f-bc77-751d89b544c7-metrics-certs\") pod \"maas-api-e2e-worker-w6-2d619f-6979556cf5-4sd4b\" (UID: \"ab4a1779-440d-442f-bc77-751d89b544c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-2d619f-6979556cf5-4sd4b"
Nov 03 22:22:18.837315 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:18.837274    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab4a1779-440d-442f-bc77-751d89b544c7-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-2d619f-6979556cf5-4sd4b\" (UID: \"ab4a1779-440d-442f-bc77-751d89b544c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-2d619f-6979556cf5-4sd4b"
Nov 03 22:22:18.837497 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:18.837369    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh2rz\" (UniqueName: \"kubernetes.io/projected/ab4a1779-440d-442f-bc77-751d89b544c7-kube-api-access-lh2rz\") pod \"maas-api-e2e-worker-w6-2d619f-6979556cf5-4sd4b\" (UID: \"ab4a1779-440d-442f-bc77-751d89b544c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-2d619f-6979556cf5-4sd4b"
Nov 03 22:22:18.837497 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:18.837414    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab4a1779-440d-442f-bc77-751d89b544c7-metrics-certs\") pod \"maas-api-e2e-worker-w6-2d619f-6979556cf5-4sd4b\" (UID: \"ab4a1779-440d-442f-bc77-751d89b544c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-2d619f-6979556cf5-4sd4b"
Nov 03 22:22:18.837497 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:18.837470    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab4a1779-440d-442f-bc77-751d89b544c7-maas-api-tls\") pod \"maas-api-e2e-worker-w6-2d619f-6979556cf5-4sd4b\" (UID: \"ab4a1779-440d-442f-bc77-751d89b544c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-2d619f-6979556cf5-4sd4b"
Nov 03 22:22:18.838081 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:18.838055    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab4a1779-440d-442f-bc77-751d89b544c7-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-2d619f-6979556cf5-4sd4b\" (UID: \"ab4a1779-440d-442f-bc77-751d89b544c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-2d619f-6979556cf5-4sd4b"
Nov 03 22:22:18.840271 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:18.840241    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab4a1779-440d-442f-bc77-751d89b544c7-metrics-certs\") pod \"maas-api-e2e-worker-w6-2d619f-6979556cf5-4sd4b\" (UID: \"ab4a1779-440d-442f-bc77-751d89b544c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-2d619f-6979556cf5-4sd4b"
Nov 03 22:22:18.840384 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:18.840330    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab4a1779-440d-442f-bc77-751d89b544c7-maas-api-tls\") pod \"maas-api-e2e-worker-w6-2d619f-6979556cf5-4sd4b\" (UID: \"ab4a1779-440d-442f-bc77-751d89b544c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-2d619f-6979556cf5-4sd4b"
Nov 03 22:22:18.861346 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:18.861314    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh2rz\" (UniqueName: \"kubernetes.io/projected/ab4a1779-440d-442f-bc77-751d89b544c7-kube-api-access-lh2rz\") pod \"maas-api-e2e-worker-w6-2d619f-6979556cf5-4sd4b\" (UID: \"ab4a1779-440d-442f-bc77-751d89b544c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-2d619f-6979556cf5-4sd4b"
Nov 03 22:22:18.977100 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:18.977068    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-2d619f-6979556cf5-4sd4b"
Nov 03 22:22:19.273452 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:19.273412    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-2d619f-6979556cf5-4sd4b"]
Nov 03 22:22:19.275792 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:22:19.275767    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab4a1779_440d_442f_bc77_751d89b544c7.slice/crio-ac95634ae6cc4471861271e7f7bc1fab91744db1a04b10d103907a3de1a3c0c4 WatchSource:0}: Error finding container ac95634ae6cc4471861271e7f7bc1fab91744db1a04b10d103907a3de1a3c0c4: Status 404 returned error can't find the container with id ac95634ae6cc4471861271e7f7bc1fab91744db1a04b10d103907a3de1a3c0c4
Nov 03 22:22:19.533300 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:19.533137    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:19.570607 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:19.570574    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-2d619f-6979556cf5-4sd4b" event={"ID":"ab4a1779-440d-442f-bc77-751d89b544c7","Type":"ContainerStarted","Data":"ac95634ae6cc4471861271e7f7bc1fab91744db1a04b10d103907a3de1a3c0c4"}
Nov 03 22:22:20.311801 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:20.311763    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:20.463411 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:20.463375    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-e19850-788cfbf7bb-jftgq"]
Nov 03 22:22:20.467121 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:20.467084    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:20.469553 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:20.469527    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e19850-788cfbf7bb-jftgq"
Nov 03 22:22:20.486149 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:20.486113    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-e19850-788cfbf7bb-jftgq"]
Nov 03 22:22:20.496344 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:20.496317    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-e19850-serving-cert\""
Nov 03 22:22:20.555662 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:20.555634    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f88957ba-3271-46ca-b439-da6a010dabe9-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-e19850-788cfbf7bb-jftgq\" (UID: \"f88957ba-3271-46ca-b439-da6a010dabe9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e19850-788cfbf7bb-jftgq"
Nov 03 22:22:20.555830 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:20.555687    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f88957ba-3271-46ca-b439-da6a010dabe9-maas-api-tls\") pod \"maas-api-e2e-worker-w1-e19850-788cfbf7bb-jftgq\" (UID: \"f88957ba-3271-46ca-b439-da6a010dabe9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e19850-788cfbf7bb-jftgq"
Nov 03 22:22:20.555830 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:20.555788    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f88957ba-3271-46ca-b439-da6a010dabe9-metrics-certs\") pod \"maas-api-e2e-worker-w1-e19850-788cfbf7bb-jftgq\" (UID: \"f88957ba-3271-46ca-b439-da6a010dabe9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e19850-788cfbf7bb-jftgq"
Nov 03 22:22:20.555830 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:20.555823    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph7nw\" (UniqueName: \"kubernetes.io/projected/f88957ba-3271-46ca-b439-da6a010dabe9-kube-api-access-ph7nw\") pod \"maas-api-e2e-worker-w1-e19850-788cfbf7bb-jftgq\" (UID: \"f88957ba-3271-46ca-b439-da6a010dabe9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e19850-788cfbf7bb-jftgq"
Nov 03 22:22:20.577664 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:20.577577    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-2d619f-6979556cf5-4sd4b" event={"ID":"ab4a1779-440d-442f-bc77-751d89b544c7","Type":"ContainerStarted","Data":"fe7b04b4e6db81ae3bb411ad59697f6bfda6e91e474ef92b1dc670f71c80521d"}
Nov 03 22:22:20.577664 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:20.577637    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-2d619f-6979556cf5-4sd4b"
Nov 03 22:22:20.609515 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:20.609482    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:20.620458 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:20.620406    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-2d619f-6979556cf5-4sd4b" podStartSLOduration=2.187935125 podStartE2EDuration="2.620390438s" totalImagesPullingTime="432.455313ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:22:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:20.619713552 +0000 UTC m=+564.645569415" watchObservedRunningTime="2026-11-03 22:22:20.620390438 +0000 UTC m=+564.646246300"
Nov 03 22:22:20.657284 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:20.657255    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f88957ba-3271-46ca-b439-da6a010dabe9-metrics-certs\") pod \"maas-api-e2e-worker-w1-e19850-788cfbf7bb-jftgq\" (UID: \"f88957ba-3271-46ca-b439-da6a010dabe9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e19850-788cfbf7bb-jftgq"
Nov 03 22:22:20.657434 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:20.657301    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph7nw\" (UniqueName: \"kubernetes.io/projected/f88957ba-3271-46ca-b439-da6a010dabe9-kube-api-access-ph7nw\") pod \"maas-api-e2e-worker-w1-e19850-788cfbf7bb-jftgq\" (UID: \"f88957ba-3271-46ca-b439-da6a010dabe9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e19850-788cfbf7bb-jftgq"
Nov 03 22:22:20.657434 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:20.657366    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f88957ba-3271-46ca-b439-da6a010dabe9-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-e19850-788cfbf7bb-jftgq\" (UID: \"f88957ba-3271-46ca-b439-da6a010dabe9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e19850-788cfbf7bb-jftgq"
Nov 03 22:22:20.657434 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:20.657425    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f88957ba-3271-46ca-b439-da6a010dabe9-maas-api-tls\") pod \"maas-api-e2e-worker-w1-e19850-788cfbf7bb-jftgq\" (UID: \"f88957ba-3271-46ca-b439-da6a010dabe9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e19850-788cfbf7bb-jftgq"
Nov 03 22:22:20.658126 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:20.658098    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f88957ba-3271-46ca-b439-da6a010dabe9-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-e19850-788cfbf7bb-jftgq\" (UID: \"f88957ba-3271-46ca-b439-da6a010dabe9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e19850-788cfbf7bb-jftgq"
Nov 03 22:22:20.659731 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:20.659705    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f88957ba-3271-46ca-b439-da6a010dabe9-maas-api-tls\") pod \"maas-api-e2e-worker-w1-e19850-788cfbf7bb-jftgq\" (UID: \"f88957ba-3271-46ca-b439-da6a010dabe9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e19850-788cfbf7bb-jftgq"
Nov 03 22:22:20.659843 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:20.659825    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f88957ba-3271-46ca-b439-da6a010dabe9-metrics-certs\") pod \"maas-api-e2e-worker-w1-e19850-788cfbf7bb-jftgq\" (UID: \"f88957ba-3271-46ca-b439-da6a010dabe9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e19850-788cfbf7bb-jftgq"
Nov 03 22:22:20.680049 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:20.680019    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph7nw\" (UniqueName: \"kubernetes.io/projected/f88957ba-3271-46ca-b439-da6a010dabe9-kube-api-access-ph7nw\") pod \"maas-api-e2e-worker-w1-e19850-788cfbf7bb-jftgq\" (UID: \"f88957ba-3271-46ca-b439-da6a010dabe9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e19850-788cfbf7bb-jftgq"
Nov 03 22:22:20.746032 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:20.745994    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:20.783143 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:20.783112    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e19850-788cfbf7bb-jftgq"
Nov 03 22:22:21.083825 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:21.083641    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-e19850-788cfbf7bb-jftgq"]
Nov 03 22:22:21.086081 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:22:21.086043    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf88957ba_3271_46ca_b439_da6a010dabe9.slice/crio-f5fe50a5e01bd9236d8d550fd6399572e2c28e122a6d41fa8343341fbbe11f35 WatchSource:0}: Error finding container f5fe50a5e01bd9236d8d550fd6399572e2c28e122a6d41fa8343341fbbe11f35: Status 404 returned error can't find the container with id f5fe50a5e01bd9236d8d550fd6399572e2c28e122a6d41fa8343341fbbe11f35
Nov 03 22:22:21.262640 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:21.262602    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:21.404772 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:21.404744    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:21.584461 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:21.584422    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e19850-788cfbf7bb-jftgq" event={"ID":"f88957ba-3271-46ca-b439-da6a010dabe9","Type":"ContainerStarted","Data":"4156b300e2f7ceec0c17c7b47875fe1331a7aebef00279d030b1fd7fd873f68b"}
Nov 03 22:22:21.584649 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:21.584480    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e19850-788cfbf7bb-jftgq" event={"ID":"f88957ba-3271-46ca-b439-da6a010dabe9","Type":"ContainerStarted","Data":"f5fe50a5e01bd9236d8d550fd6399572e2c28e122a6d41fa8343341fbbe11f35"}
Nov 03 22:22:21.627107 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:21.627057    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e19850-788cfbf7bb-jftgq" podStartSLOduration=1.335115113 podStartE2EDuration="1.62704194s" totalImagesPullingTime="291.926827ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:22:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:21.625509732 +0000 UTC m=+565.651365596" watchObservedRunningTime="2026-11-03 22:22:21.62704194 +0000 UTC m=+565.652897813"
Nov 03 22:22:22.166409 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:22.166377    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-434ce3-f9c44bc7b-mb2rs"]
Nov 03 22:22:22.170048 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:22.170027    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-434ce3-f9c44bc7b-mb2rs"
Nov 03 22:22:22.187429 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:22.187406    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-434ce3-serving-cert\""
Nov 03 22:22:22.202779 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:22.202745    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-434ce3-f9c44bc7b-mb2rs"]
Nov 03 22:22:22.273036 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:22.273000    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f78011a-db80-4c8f-b562-a80b6970d207-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-434ce3-f9c44bc7b-mb2rs\" (UID: \"0f78011a-db80-4c8f-b562-a80b6970d207\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-434ce3-f9c44bc7b-mb2rs"
Nov 03 22:22:22.273241 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:22.273059    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f78011a-db80-4c8f-b562-a80b6970d207-maas-api-tls\") pod \"maas-api-e2e-worker-w5-434ce3-f9c44bc7b-mb2rs\" (UID: \"0f78011a-db80-4c8f-b562-a80b6970d207\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-434ce3-f9c44bc7b-mb2rs"
Nov 03 22:22:22.273241 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:22.273174    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svpzs\" (UniqueName: \"kubernetes.io/projected/0f78011a-db80-4c8f-b562-a80b6970d207-kube-api-access-svpzs\") pod \"maas-api-e2e-worker-w5-434ce3-f9c44bc7b-mb2rs\" (UID: \"0f78011a-db80-4c8f-b562-a80b6970d207\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-434ce3-f9c44bc7b-mb2rs"
Nov 03 22:22:22.273334 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:22.273249    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f78011a-db80-4c8f-b562-a80b6970d207-metrics-certs\") pod \"maas-api-e2e-worker-w5-434ce3-f9c44bc7b-mb2rs\" (UID: \"0f78011a-db80-4c8f-b562-a80b6970d207\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-434ce3-f9c44bc7b-mb2rs"
Nov 03 22:22:22.296539 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:22.296505    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:22.374717 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:22.374686    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f78011a-db80-4c8f-b562-a80b6970d207-metrics-certs\") pod \"maas-api-e2e-worker-w5-434ce3-f9c44bc7b-mb2rs\" (UID: \"0f78011a-db80-4c8f-b562-a80b6970d207\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-434ce3-f9c44bc7b-mb2rs"
Nov 03 22:22:22.374937 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:22.374749    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f78011a-db80-4c8f-b562-a80b6970d207-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-434ce3-f9c44bc7b-mb2rs\" (UID: \"0f78011a-db80-4c8f-b562-a80b6970d207\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-434ce3-f9c44bc7b-mb2rs"
Nov 03 22:22:22.374937 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:22.374787    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f78011a-db80-4c8f-b562-a80b6970d207-maas-api-tls\") pod \"maas-api-e2e-worker-w5-434ce3-f9c44bc7b-mb2rs\" (UID: \"0f78011a-db80-4c8f-b562-a80b6970d207\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-434ce3-f9c44bc7b-mb2rs"
Nov 03 22:22:22.374937 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:22.374825    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svpzs\" (UniqueName: \"kubernetes.io/projected/0f78011a-db80-4c8f-b562-a80b6970d207-kube-api-access-svpzs\") pod \"maas-api-e2e-worker-w5-434ce3-f9c44bc7b-mb2rs\" (UID: \"0f78011a-db80-4c8f-b562-a80b6970d207\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-434ce3-f9c44bc7b-mb2rs"
Nov 03 22:22:22.375603 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:22.375581    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f78011a-db80-4c8f-b562-a80b6970d207-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-434ce3-f9c44bc7b-mb2rs\" (UID: \"0f78011a-db80-4c8f-b562-a80b6970d207\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-434ce3-f9c44bc7b-mb2rs"
Nov 03 22:22:22.377254 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:22.377207    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f78011a-db80-4c8f-b562-a80b6970d207-metrics-certs\") pod \"maas-api-e2e-worker-w5-434ce3-f9c44bc7b-mb2rs\" (UID: \"0f78011a-db80-4c8f-b562-a80b6970d207\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-434ce3-f9c44bc7b-mb2rs"
Nov 03 22:22:22.377342 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:22.377326    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f78011a-db80-4c8f-b562-a80b6970d207-maas-api-tls\") pod \"maas-api-e2e-worker-w5-434ce3-f9c44bc7b-mb2rs\" (UID: \"0f78011a-db80-4c8f-b562-a80b6970d207\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-434ce3-f9c44bc7b-mb2rs"
Nov 03 22:22:22.398821 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:22.398794    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svpzs\" (UniqueName: \"kubernetes.io/projected/0f78011a-db80-4c8f-b562-a80b6970d207-kube-api-access-svpzs\") pod \"maas-api-e2e-worker-w5-434ce3-f9c44bc7b-mb2rs\" (UID: \"0f78011a-db80-4c8f-b562-a80b6970d207\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-434ce3-f9c44bc7b-mb2rs"
Nov 03 22:22:22.435148 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:22.435071    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:22.482390 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:22.482356    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-434ce3-f9c44bc7b-mb2rs"
Nov 03 22:22:22.575729 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:22.575693    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:22.589756 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:22.589728    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e19850-788cfbf7bb-jftgq"
Nov 03 22:22:22.719784 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:22.719701    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:22.784937 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:22.784911    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-434ce3-f9c44bc7b-mb2rs"]
Nov 03 22:22:22.787187 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:22:22.787160    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f78011a_db80_4c8f_b562_a80b6970d207.slice/crio-2eacc6da03ccd38d2ff738cf62e3980f4c603bdaf81b68f8e5254c7d43a70bd3 WatchSource:0}: Error finding container 2eacc6da03ccd38d2ff738cf62e3980f4c603bdaf81b68f8e5254c7d43a70bd3: Status 404 returned error can't find the container with id 2eacc6da03ccd38d2ff738cf62e3980f4c603bdaf81b68f8e5254c7d43a70bd3
Nov 03 22:22:22.857892 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:22.857858    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:23.597429 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:23.597389    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-434ce3-f9c44bc7b-mb2rs" event={"ID":"0f78011a-db80-4c8f-b562-a80b6970d207","Type":"ContainerStarted","Data":"5bb4b0ecdae28fbccd17193cfbeadcd3ece64a89bd7dec712168772a0a12453a"}
Nov 03 22:22:23.597864 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:23.597446    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-434ce3-f9c44bc7b-mb2rs" event={"ID":"0f78011a-db80-4c8f-b562-a80b6970d207","Type":"ContainerStarted","Data":"2eacc6da03ccd38d2ff738cf62e3980f4c603bdaf81b68f8e5254c7d43a70bd3"}
Nov 03 22:22:23.647099 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:23.647039    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-434ce3-f9c44bc7b-mb2rs" podStartSLOduration=1.348822 podStartE2EDuration="1.647022977s" totalImagesPullingTime="298.200977ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:22:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:23.645742754 +0000 UTC m=+567.671598615" watchObservedRunningTime="2026-11-03 22:22:23.647022977 +0000 UTC m=+567.672878841"
Nov 03 22:22:24.062388 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:24.062352    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:24.200479 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:24.200435    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:24.349494 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:24.349414    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:24.489936 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:24.489885    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:24.602504 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:24.602413    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-434ce3-f9c44bc7b-mb2rs"
Nov 03 22:22:25.209499 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:25.209465    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:25.355959 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:25.355920    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:25.824860 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:25.824831    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:26.590598 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:26.590569    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-2d619f-6979556cf5-4sd4b"
Nov 03 22:22:27.369310 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:27.369279    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-88a4d9-6bb4875fbc-xrztv"]
Nov 03 22:22:27.372772 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:27.372755    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-88a4d9-6bb4875fbc-xrztv"
Nov 03 22:22:27.399865 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:27.399839    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-88a4d9-serving-cert\""
Nov 03 22:22:27.402923 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:27.402894    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-88a4d9-6bb4875fbc-xrztv"]
Nov 03 22:22:27.420916 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:27.420879    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkgj7\" (UniqueName: \"kubernetes.io/projected/11f1ee81-e4ca-4096-bf74-62018777d965-kube-api-access-qkgj7\") pod \"maas-api-e2e-worker-w0-88a4d9-6bb4875fbc-xrztv\" (UID: \"11f1ee81-e4ca-4096-bf74-62018777d965\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-88a4d9-6bb4875fbc-xrztv"
Nov 03 22:22:27.421073 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:27.420995    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11f1ee81-e4ca-4096-bf74-62018777d965-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-88a4d9-6bb4875fbc-xrztv\" (UID: \"11f1ee81-e4ca-4096-bf74-62018777d965\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-88a4d9-6bb4875fbc-xrztv"
Nov 03 22:22:27.421139 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:27.421126    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11f1ee81-e4ca-4096-bf74-62018777d965-maas-api-tls\") pod \"maas-api-e2e-worker-w0-88a4d9-6bb4875fbc-xrztv\" (UID: \"11f1ee81-e4ca-4096-bf74-62018777d965\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-88a4d9-6bb4875fbc-xrztv"
Nov 03 22:22:27.421194 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:27.421159    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11f1ee81-e4ca-4096-bf74-62018777d965-metrics-certs\") pod \"maas-api-e2e-worker-w0-88a4d9-6bb4875fbc-xrztv\" (UID: \"11f1ee81-e4ca-4096-bf74-62018777d965\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-88a4d9-6bb4875fbc-xrztv"
Nov 03 22:22:27.506375 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:27.506341    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:27.522544 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:27.522518    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11f1ee81-e4ca-4096-bf74-62018777d965-maas-api-tls\") pod \"maas-api-e2e-worker-w0-88a4d9-6bb4875fbc-xrztv\" (UID: \"11f1ee81-e4ca-4096-bf74-62018777d965\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-88a4d9-6bb4875fbc-xrztv"
Nov 03 22:22:27.522673 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:27.522548    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11f1ee81-e4ca-4096-bf74-62018777d965-metrics-certs\") pod \"maas-api-e2e-worker-w0-88a4d9-6bb4875fbc-xrztv\" (UID: \"11f1ee81-e4ca-4096-bf74-62018777d965\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-88a4d9-6bb4875fbc-xrztv"
Nov 03 22:22:27.522673 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:27.522588    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkgj7\" (UniqueName: \"kubernetes.io/projected/11f1ee81-e4ca-4096-bf74-62018777d965-kube-api-access-qkgj7\") pod \"maas-api-e2e-worker-w0-88a4d9-6bb4875fbc-xrztv\" (UID: \"11f1ee81-e4ca-4096-bf74-62018777d965\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-88a4d9-6bb4875fbc-xrztv"
Nov 03 22:22:27.522673 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:27.522647    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11f1ee81-e4ca-4096-bf74-62018777d965-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-88a4d9-6bb4875fbc-xrztv\" (UID: \"11f1ee81-e4ca-4096-bf74-62018777d965\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-88a4d9-6bb4875fbc-xrztv"
Nov 03 22:22:27.522836 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:22:27.522740    4431 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-88a4d9-serving-cert: secret "maas-api-e2e-worker-w0-88a4d9-serving-cert" not found
Nov 03 22:22:27.522836 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:22:27.522804    4431 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11f1ee81-e4ca-4096-bf74-62018777d965-maas-api-tls podName:11f1ee81-e4ca-4096-bf74-62018777d965 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:28.022788632 +0000 UTC m=+572.048644475 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/11f1ee81-e4ca-4096-bf74-62018777d965-maas-api-tls") pod "maas-api-e2e-worker-w0-88a4d9-6bb4875fbc-xrztv" (UID: "11f1ee81-e4ca-4096-bf74-62018777d965") : secret "maas-api-e2e-worker-w0-88a4d9-serving-cert" not found
Nov 03 22:22:27.523382 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:27.523362    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11f1ee81-e4ca-4096-bf74-62018777d965-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-88a4d9-6bb4875fbc-xrztv\" (UID: \"11f1ee81-e4ca-4096-bf74-62018777d965\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-88a4d9-6bb4875fbc-xrztv"
Nov 03 22:22:27.524987 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:27.524965    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11f1ee81-e4ca-4096-bf74-62018777d965-metrics-certs\") pod \"maas-api-e2e-worker-w0-88a4d9-6bb4875fbc-xrztv\" (UID: \"11f1ee81-e4ca-4096-bf74-62018777d965\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-88a4d9-6bb4875fbc-xrztv"
Nov 03 22:22:27.546727 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:27.546703    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkgj7\" (UniqueName: \"kubernetes.io/projected/11f1ee81-e4ca-4096-bf74-62018777d965-kube-api-access-qkgj7\") pod \"maas-api-e2e-worker-w0-88a4d9-6bb4875fbc-xrztv\" (UID: \"11f1ee81-e4ca-4096-bf74-62018777d965\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-88a4d9-6bb4875fbc-xrztv"
Nov 03 22:22:27.648718 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:27.648627    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:27.794010 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:27.793976    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:28.028088 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:28.028037    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11f1ee81-e4ca-4096-bf74-62018777d965-maas-api-tls\") pod \"maas-api-e2e-worker-w0-88a4d9-6bb4875fbc-xrztv\" (UID: \"11f1ee81-e4ca-4096-bf74-62018777d965\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-88a4d9-6bb4875fbc-xrztv"
Nov 03 22:22:28.030545 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:28.030518    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11f1ee81-e4ca-4096-bf74-62018777d965-maas-api-tls\") pod \"maas-api-e2e-worker-w0-88a4d9-6bb4875fbc-xrztv\" (UID: \"11f1ee81-e4ca-4096-bf74-62018777d965\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-88a4d9-6bb4875fbc-xrztv"
Nov 03 22:22:28.285581 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:28.285501    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-88a4d9-6bb4875fbc-xrztv"
Nov 03 22:22:28.587429 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:28.587357    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-88a4d9-6bb4875fbc-xrztv"]
Nov 03 22:22:28.589812 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:22:28.589784    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11f1ee81_e4ca_4096_bf74_62018777d965.slice/crio-bda9f16f5baddd4053d2e36eaea5bed547982e02e5d13714f0ca78c3efbc0508 WatchSource:0}: Error finding container bda9f16f5baddd4053d2e36eaea5bed547982e02e5d13714f0ca78c3efbc0508: Status 404 returned error can't find the container with id bda9f16f5baddd4053d2e36eaea5bed547982e02e5d13714f0ca78c3efbc0508
Nov 03 22:22:28.602820 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:28.602795    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e19850-788cfbf7bb-jftgq"
Nov 03 22:22:28.620456 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:28.620431    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-88a4d9-6bb4875fbc-xrztv" event={"ID":"11f1ee81-e4ca-4096-bf74-62018777d965","Type":"ContainerStarted","Data":"bda9f16f5baddd4053d2e36eaea5bed547982e02e5d13714f0ca78c3efbc0508"}
Nov 03 22:22:29.299363 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:29.299285    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-50d738-75496c474b-xjvd2"]
Nov 03 22:22:29.303166 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:29.303142    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-50d738-75496c474b-xjvd2"
Nov 03 22:22:29.323920 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:29.323873    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-50d738-serving-cert\""
Nov 03 22:22:29.335197 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:29.335049    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-50d738-75496c474b-xjvd2"]
Nov 03 22:22:29.341863 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:29.341828    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqpb8\" (UniqueName: \"kubernetes.io/projected/0696d31c-b588-4817-ba3b-be9d9591eef9-kube-api-access-mqpb8\") pod \"maas-api-e2e-worker-w4-50d738-75496c474b-xjvd2\" (UID: \"0696d31c-b588-4817-ba3b-be9d9591eef9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-50d738-75496c474b-xjvd2"
Nov 03 22:22:29.342105 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:29.342070    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0696d31c-b588-4817-ba3b-be9d9591eef9-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-50d738-75496c474b-xjvd2\" (UID: \"0696d31c-b588-4817-ba3b-be9d9591eef9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-50d738-75496c474b-xjvd2"
Nov 03 22:22:29.342205 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:29.342187    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0696d31c-b588-4817-ba3b-be9d9591eef9-maas-api-tls\") pod \"maas-api-e2e-worker-w4-50d738-75496c474b-xjvd2\" (UID: \"0696d31c-b588-4817-ba3b-be9d9591eef9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-50d738-75496c474b-xjvd2"
Nov 03 22:22:29.342333 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:29.342313    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0696d31c-b588-4817-ba3b-be9d9591eef9-metrics-certs\") pod \"maas-api-e2e-worker-w4-50d738-75496c474b-xjvd2\" (UID: \"0696d31c-b588-4817-ba3b-be9d9591eef9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-50d738-75496c474b-xjvd2"
Nov 03 22:22:29.427783 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:29.427744    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:29.463735 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:29.463692    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0696d31c-b588-4817-ba3b-be9d9591eef9-metrics-certs\") pod \"maas-api-e2e-worker-w4-50d738-75496c474b-xjvd2\" (UID: \"0696d31c-b588-4817-ba3b-be9d9591eef9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-50d738-75496c474b-xjvd2"
Nov 03 22:22:29.463940 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:29.463757    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqpb8\" (UniqueName: \"kubernetes.io/projected/0696d31c-b588-4817-ba3b-be9d9591eef9-kube-api-access-mqpb8\") pod \"maas-api-e2e-worker-w4-50d738-75496c474b-xjvd2\" (UID: \"0696d31c-b588-4817-ba3b-be9d9591eef9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-50d738-75496c474b-xjvd2"
Nov 03 22:22:29.463940 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:29.463850    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0696d31c-b588-4817-ba3b-be9d9591eef9-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-50d738-75496c474b-xjvd2\" (UID: \"0696d31c-b588-4817-ba3b-be9d9591eef9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-50d738-75496c474b-xjvd2"
Nov 03 22:22:29.463940 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:29.463897    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0696d31c-b588-4817-ba3b-be9d9591eef9-maas-api-tls\") pod \"maas-api-e2e-worker-w4-50d738-75496c474b-xjvd2\" (UID: \"0696d31c-b588-4817-ba3b-be9d9591eef9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-50d738-75496c474b-xjvd2"
Nov 03 22:22:29.464616 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:29.464588    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0696d31c-b588-4817-ba3b-be9d9591eef9-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-50d738-75496c474b-xjvd2\" (UID: \"0696d31c-b588-4817-ba3b-be9d9591eef9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-50d738-75496c474b-xjvd2"
Nov 03 22:22:29.466345 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:29.466316    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0696d31c-b588-4817-ba3b-be9d9591eef9-metrics-certs\") pod \"maas-api-e2e-worker-w4-50d738-75496c474b-xjvd2\" (UID: \"0696d31c-b588-4817-ba3b-be9d9591eef9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-50d738-75496c474b-xjvd2"
Nov 03 22:22:29.466440 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:29.466358    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0696d31c-b588-4817-ba3b-be9d9591eef9-maas-api-tls\") pod \"maas-api-e2e-worker-w4-50d738-75496c474b-xjvd2\" (UID: \"0696d31c-b588-4817-ba3b-be9d9591eef9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-50d738-75496c474b-xjvd2"
Nov 03 22:22:29.487066 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:29.487031    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqpb8\" (UniqueName: \"kubernetes.io/projected/0696d31c-b588-4817-ba3b-be9d9591eef9-kube-api-access-mqpb8\") pod \"maas-api-e2e-worker-w4-50d738-75496c474b-xjvd2\" (UID: \"0696d31c-b588-4817-ba3b-be9d9591eef9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-50d738-75496c474b-xjvd2"
Nov 03 22:22:29.617288 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:29.617171    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-50d738-75496c474b-xjvd2"
Nov 03 22:22:29.626897 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:29.626867    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-88a4d9-6bb4875fbc-xrztv" event={"ID":"11f1ee81-e4ca-4096-bf74-62018777d965","Type":"ContainerStarted","Data":"667b577c3c922122f3f20ee6d8cbbcde6396484ee5cb9c0ae72518b4ea47b3ce"}
Nov 03 22:22:29.627047 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:29.627026    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-88a4d9-6bb4875fbc-xrztv"
Nov 03 22:22:29.664029 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:29.663989    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:29.680902 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:29.680845    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-88a4d9-6bb4875fbc-xrztv" podStartSLOduration=2.303439535 podStartE2EDuration="2.680827219s" totalImagesPullingTime="377.387684ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:22:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:29.678760536 +0000 UTC m=+573.704616402" watchObservedRunningTime="2026-11-03 22:22:29.680827219 +0000 UTC m=+573.706683080"
Nov 03 22:22:29.815662 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:29.815625    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:29.953267 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:29.947558    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-50d738-75496c474b-xjvd2"]
Nov 03 22:22:29.966906 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:22:29.966863    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0696d31c_b588_4817_ba3b_be9d9591eef9.slice/crio-fdf31988aefc04513129196be1db4ae3ecf3dd2ce5306587a84e3d1acaec8f12 WatchSource:0}: Error finding container fdf31988aefc04513129196be1db4ae3ecf3dd2ce5306587a84e3d1acaec8f12: Status 404 returned error can't find the container with id fdf31988aefc04513129196be1db4ae3ecf3dd2ce5306587a84e3d1acaec8f12
Nov 03 22:22:30.612943 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:30.612857    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-434ce3-f9c44bc7b-mb2rs"
Nov 03 22:22:30.632665 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:30.632630    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-50d738-75496c474b-xjvd2" event={"ID":"0696d31c-b588-4817-ba3b-be9d9591eef9","Type":"ContainerStarted","Data":"9fce25520a3806c5fce2b8c50b923ce80125e4cc0b7d2be77fe0288ce33cbb7a"}
Nov 03 22:22:30.633053 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:30.632684    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-50d738-75496c474b-xjvd2" event={"ID":"0696d31c-b588-4817-ba3b-be9d9591eef9","Type":"ContainerStarted","Data":"fdf31988aefc04513129196be1db4ae3ecf3dd2ce5306587a84e3d1acaec8f12"}
Nov 03 22:22:30.633053 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:30.632803    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-50d738-75496c474b-xjvd2"
Nov 03 22:22:30.707688 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:30.707637    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-50d738-75496c474b-xjvd2" podStartSLOduration=1.374953746 podStartE2EDuration="1.707622121s" totalImagesPullingTime="332.668375ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:22:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:30.704944123 +0000 UTC m=+574.730799987" watchObservedRunningTime="2026-11-03 22:22:30.707622121 +0000 UTC m=+574.733478032"
Nov 03 22:22:31.284273 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:22:31.284235    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:22:31.366832 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:31.366798    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:33.025950 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:33.025915    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:33.171196 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:33.171164    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:33.320778 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:33.320697    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:35.052570 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:35.052517    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:35.620302 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:35.620262    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:35.638093 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:35.638066    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-88a4d9-6bb4875fbc-xrztv"
Nov 03 22:22:36.310548 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:36.310514    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:36.455986 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:36.455948    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:36.643210 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:36.643131    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-50d738-75496c474b-xjvd2"
Nov 03 22:22:37.097460 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:37.097424    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:38.253406 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:38.253363    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:38.690059 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:38.690023    4431 generic.go:356] "Generic (PLEG): container finished" podID="b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c" containerID="be9f70a522fa1e9bb79e5fde7dfc3cce48574a605cdbf375d5d5807d2e025250" exitCode=0
Nov 03 22:22:38.690259 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:38.690097    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96" event={"ID":"b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c","Type":"ContainerDied","Data":"be9f70a522fa1e9bb79e5fde7dfc3cce48574a605cdbf375d5d5807d2e025250"}
Nov 03 22:22:38.690519 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:38.690506    4431 scope.go:122] "RemoveContainer" containerID="be9f70a522fa1e9bb79e5fde7dfc3cce48574a605cdbf375d5d5807d2e025250"
Nov 03 22:22:38.691928 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:38.691898    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:22:38.692063 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:38.691983    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:22:38.692063 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:38.692032    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:22:39.023685 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:39.023599    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:39.696791 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:39.696757    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96" event={"ID":"b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c","Type":"ContainerStarted","Data":"dc025a25fa12213e4c7c4dab4d84697325cbbcb38eaffcd5ce67a67128d8dae6"}
Nov 03 22:22:40.102036 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:40.101910    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96"
Nov 03 22:22:40.103101 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:40.103073    4431 patch_prober.go:29] interesting pod/e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Nov 03 22:22:40.103195 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:40.103131    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96" podUID="b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Nov 03 22:22:40.904845 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:40.904801    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:41.102338 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:41.102302    4431 patch_prober.go:29] interesting pod/e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Nov 03 22:22:41.102521 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:41.102353    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96" podUID="b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Nov 03 22:22:41.120813 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:41.120779    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:41.820053 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:41.820020    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:41.989447 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:41.989411    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:42.102335 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:42.102256    4431 patch_prober.go:29] interesting pod/e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Nov 03 22:22:42.102480 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:42.102337    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96" podUID="b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Nov 03 22:22:42.317982 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:42.317947    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:42.495675 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:42.495639    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:42.662240 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:42.661821    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:43.102630 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:43.102593    4431 patch_prober.go:29] interesting pod/e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Nov 03 22:22:43.103070 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:43.103046    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96" podUID="b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Nov 03 22:22:43.188988 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:43.188945    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:44.102296 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:44.102263    4431 patch_prober.go:29] interesting pod/e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Nov 03 22:22:44.102473 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:44.102314    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96" podUID="b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Nov 03 22:22:44.552239 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:44.545320    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96"
Nov 03 22:22:44.936849 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:44.936822    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:45.081351 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:45.081321    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:45.106402 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:45.106375    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96"
Nov 03 22:22:45.107156 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:45.107141    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96"
Nov 03 22:22:46.709286 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:46.709248    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:46.898787 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:46.898742    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:47.463882 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:47.463851    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:47.607504 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:47.607471    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:47.752996 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:47.752903    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:48.322618 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:48.322582    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:48.983604 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:48.983563    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:50.130883 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:50.130845    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:50.922823 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:50.922792    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:51.490752 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:51.490721    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:51.631413 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:51.631381    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:51.769246 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:51.769143    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:51.914349 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:51.914314    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:52.518965 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:52.518930    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:52.665993 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:52.665964    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:53.493023 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:53.492990    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:54.079551 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:54.079503    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:55.222339 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:55.222304    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:56.155326 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:56.155286    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mx5q8_ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea/ovn-acl-logging/0.log"
Nov 03 22:22:56.156659 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:56.156632    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mx5q8_ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea/ovn-acl-logging/0.log"
Nov 03 22:22:56.817331 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:56.817289    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:58.561958 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:58.561925    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-88a4d9/simulator-w0-88a4d9-kserve-785f84888f-sjbzp"]
Nov 03 22:22:58.565400 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:58.565379    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-88a4d9/simulator-w0-88a4d9-kserve-785f84888f-sjbzp"
Nov 03 22:22:58.590639 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:58.590606    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18bd460b-a0f1-415c-bcb4-41c9f70d2926-model-cache\") pod \"simulator-w0-88a4d9-kserve-785f84888f-sjbzp\" (UID: \"18bd460b-a0f1-415c-bcb4-41c9f70d2926\") " pod="e2e-models-e2e-worker-w0-88a4d9/simulator-w0-88a4d9-kserve-785f84888f-sjbzp"
Nov 03 22:22:58.590639 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:58.590647    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmbrj\" (UniqueName: \"kubernetes.io/projected/18bd460b-a0f1-415c-bcb4-41c9f70d2926-kube-api-access-jmbrj\") pod \"simulator-w0-88a4d9-kserve-785f84888f-sjbzp\" (UID: \"18bd460b-a0f1-415c-bcb4-41c9f70d2926\") " pod="e2e-models-e2e-worker-w0-88a4d9/simulator-w0-88a4d9-kserve-785f84888f-sjbzp"
Nov 03 22:22:58.590845 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:58.590793    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18bd460b-a0f1-415c-bcb4-41c9f70d2926-home\") pod \"simulator-w0-88a4d9-kserve-785f84888f-sjbzp\" (UID: \"18bd460b-a0f1-415c-bcb4-41c9f70d2926\") " pod="e2e-models-e2e-worker-w0-88a4d9/simulator-w0-88a4d9-kserve-785f84888f-sjbzp"
Nov 03 22:22:58.590845 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:58.590830    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18bd460b-a0f1-415c-bcb4-41c9f70d2926-dshm\") pod \"simulator-w0-88a4d9-kserve-785f84888f-sjbzp\" (UID: \"18bd460b-a0f1-415c-bcb4-41c9f70d2926\") " pod="e2e-models-e2e-worker-w0-88a4d9/simulator-w0-88a4d9-kserve-785f84888f-sjbzp"
Nov 03 22:22:58.590988 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:58.590962    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18bd460b-a0f1-415c-bcb4-41c9f70d2926-tmp-dir\") pod \"simulator-w0-88a4d9-kserve-785f84888f-sjbzp\" (UID: \"18bd460b-a0f1-415c-bcb4-41c9f70d2926\") " pod="e2e-models-e2e-worker-w0-88a4d9/simulator-w0-88a4d9-kserve-785f84888f-sjbzp"
Nov 03 22:22:58.591042 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:58.591023    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18bd460b-a0f1-415c-bcb4-41c9f70d2926-tls-certs\") pod \"simulator-w0-88a4d9-kserve-785f84888f-sjbzp\" (UID: \"18bd460b-a0f1-415c-bcb4-41c9f70d2926\") " pod="e2e-models-e2e-worker-w0-88a4d9/simulator-w0-88a4d9-kserve-785f84888f-sjbzp"
Nov 03 22:22:58.595321 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:58.595296    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-88a4d9\"/\"openshift-service-ca.crt\""
Nov 03 22:22:58.595496 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:58.595474    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-88a4d9\"/\"kube-root-ca.crt\""
Nov 03 22:22:58.595852 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:58.595831    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-88a4d9\"/\"simulator-w0-88a4d9-kserve-self-signed-certs\""
Nov 03 22:22:58.596356 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:58.596340    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-88a4d9\"/\"default-dockercfg-kw52f\""
Nov 03 22:22:58.597500 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:58.597480    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-88a4d9/simulator-w0-88a4d9-kserve-785f84888f-sjbzp"]
Nov 03 22:22:58.692710 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:58.692669    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18bd460b-a0f1-415c-bcb4-41c9f70d2926-model-cache\") pod \"simulator-w0-88a4d9-kserve-785f84888f-sjbzp\" (UID: \"18bd460b-a0f1-415c-bcb4-41c9f70d2926\") " pod="e2e-models-e2e-worker-w0-88a4d9/simulator-w0-88a4d9-kserve-785f84888f-sjbzp"
Nov 03 22:22:58.692710 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:58.692704    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmbrj\" (UniqueName: \"kubernetes.io/projected/18bd460b-a0f1-415c-bcb4-41c9f70d2926-kube-api-access-jmbrj\") pod \"simulator-w0-88a4d9-kserve-785f84888f-sjbzp\" (UID: \"18bd460b-a0f1-415c-bcb4-41c9f70d2926\") " pod="e2e-models-e2e-worker-w0-88a4d9/simulator-w0-88a4d9-kserve-785f84888f-sjbzp"
Nov 03 22:22:58.692957 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:58.692749    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18bd460b-a0f1-415c-bcb4-41c9f70d2926-home\") pod \"simulator-w0-88a4d9-kserve-785f84888f-sjbzp\" (UID: \"18bd460b-a0f1-415c-bcb4-41c9f70d2926\") " pod="e2e-models-e2e-worker-w0-88a4d9/simulator-w0-88a4d9-kserve-785f84888f-sjbzp"
Nov 03 22:22:58.692957 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:58.692768    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18bd460b-a0f1-415c-bcb4-41c9f70d2926-dshm\") pod \"simulator-w0-88a4d9-kserve-785f84888f-sjbzp\" (UID: \"18bd460b-a0f1-415c-bcb4-41c9f70d2926\") " pod="e2e-models-e2e-worker-w0-88a4d9/simulator-w0-88a4d9-kserve-785f84888f-sjbzp"
Nov 03 22:22:58.692957 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:58.692806    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18bd460b-a0f1-415c-bcb4-41c9f70d2926-tmp-dir\") pod \"simulator-w0-88a4d9-kserve-785f84888f-sjbzp\" (UID: \"18bd460b-a0f1-415c-bcb4-41c9f70d2926\") " pod="e2e-models-e2e-worker-w0-88a4d9/simulator-w0-88a4d9-kserve-785f84888f-sjbzp"
Nov 03 22:22:58.692957 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:58.692841    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18bd460b-a0f1-415c-bcb4-41c9f70d2926-tls-certs\") pod \"simulator-w0-88a4d9-kserve-785f84888f-sjbzp\" (UID: \"18bd460b-a0f1-415c-bcb4-41c9f70d2926\") " pod="e2e-models-e2e-worker-w0-88a4d9/simulator-w0-88a4d9-kserve-785f84888f-sjbzp"
Nov 03 22:22:58.693187 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:58.693126    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18bd460b-a0f1-415c-bcb4-41c9f70d2926-model-cache\") pod \"simulator-w0-88a4d9-kserve-785f84888f-sjbzp\" (UID: \"18bd460b-a0f1-415c-bcb4-41c9f70d2926\") " pod="e2e-models-e2e-worker-w0-88a4d9/simulator-w0-88a4d9-kserve-785f84888f-sjbzp"
Nov 03 22:22:58.693187 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:58.693167    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18bd460b-a0f1-415c-bcb4-41c9f70d2926-home\") pod \"simulator-w0-88a4d9-kserve-785f84888f-sjbzp\" (UID: \"18bd460b-a0f1-415c-bcb4-41c9f70d2926\") " pod="e2e-models-e2e-worker-w0-88a4d9/simulator-w0-88a4d9-kserve-785f84888f-sjbzp"
Nov 03 22:22:58.693462 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:58.693435    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18bd460b-a0f1-415c-bcb4-41c9f70d2926-tmp-dir\") pod \"simulator-w0-88a4d9-kserve-785f84888f-sjbzp\" (UID: \"18bd460b-a0f1-415c-bcb4-41c9f70d2926\") " pod="e2e-models-e2e-worker-w0-88a4d9/simulator-w0-88a4d9-kserve-785f84888f-sjbzp"
Nov 03 22:22:58.695087 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:58.695065    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18bd460b-a0f1-415c-bcb4-41c9f70d2926-dshm\") pod \"simulator-w0-88a4d9-kserve-785f84888f-sjbzp\" (UID: \"18bd460b-a0f1-415c-bcb4-41c9f70d2926\") " pod="e2e-models-e2e-worker-w0-88a4d9/simulator-w0-88a4d9-kserve-785f84888f-sjbzp"
Nov 03 22:22:58.695265 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:58.695248    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18bd460b-a0f1-415c-bcb4-41c9f70d2926-tls-certs\") pod \"simulator-w0-88a4d9-kserve-785f84888f-sjbzp\" (UID: \"18bd460b-a0f1-415c-bcb4-41c9f70d2926\") " pod="e2e-models-e2e-worker-w0-88a4d9/simulator-w0-88a4d9-kserve-785f84888f-sjbzp"
Nov 03 22:22:58.726669 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:58.726635    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmbrj\" (UniqueName: \"kubernetes.io/projected/18bd460b-a0f1-415c-bcb4-41c9f70d2926-kube-api-access-jmbrj\") pod \"simulator-w0-88a4d9-kserve-785f84888f-sjbzp\" (UID: \"18bd460b-a0f1-415c-bcb4-41c9f70d2926\") " pod="e2e-models-e2e-worker-w0-88a4d9/simulator-w0-88a4d9-kserve-785f84888f-sjbzp"
Nov 03 22:22:58.781631 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:58.781595    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:58.877039 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:58.876956    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-88a4d9/simulator-w0-88a4d9-kserve-785f84888f-sjbzp"
Nov 03 22:22:58.933711 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:58.932835    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:22:59.189519 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:59.189486    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-88a4d9/simulator-w0-88a4d9-kserve-785f84888f-sjbzp"]
Nov 03 22:22:59.190382 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:22:59.190358    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod18bd460b_a0f1_415c_bcb4_41c9f70d2926.slice/crio-41c518a53ca4870bbf7e9db66c21461c685199dc220ae1b28a26f0df78ffa859 WatchSource:0}: Error finding container 41c518a53ca4870bbf7e9db66c21461c685199dc220ae1b28a26f0df78ffa859: Status 404 returned error can't find the container with id 41c518a53ca4870bbf7e9db66c21461c685199dc220ae1b28a26f0df78ffa859
Nov 03 22:22:59.801558 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:22:59.801526    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-88a4d9/simulator-w0-88a4d9-kserve-785f84888f-sjbzp" event={"ID":"18bd460b-a0f1-415c-bcb4-41c9f70d2926","Type":"ContainerStarted","Data":"41c518a53ca4870bbf7e9db66c21461c685199dc220ae1b28a26f0df78ffa859"}
Nov 03 22:23:00.400324 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:00.400291    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:23:00.807025 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:00.806987    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-88a4d9/simulator-w0-88a4d9-kserve-785f84888f-sjbzp" event={"ID":"18bd460b-a0f1-415c-bcb4-41c9f70d2926","Type":"ContainerStarted","Data":"bf12d317e70bd7804623af7d868541932ca2fc2a29807d96e123bc699d20a7c3"}
Nov 03 22:23:00.849835 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:00.849784    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-88a4d9/simulator-w0-88a4d9-kserve-785f84888f-sjbzp" podStartSLOduration=1.809006074 podStartE2EDuration="2.849769899s" totalImagesPullingTime="1.040763825s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:22:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:23:00.849193192 +0000 UTC m=+604.875049056" watchObservedRunningTime="2026-11-03 22:23:00.849769899 +0000 UTC m=+604.875625769"
Nov 03 22:23:00.877369 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:00.877324    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-88a4d9/simulator-w0-88a4d9-kserve-785f84888f-sjbzp"
Nov 03 22:23:01.258111 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:23:01.258071    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:23:01.904888 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:01.904855    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:23:03.549967 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:03.549928    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:23:04.905546 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:04.905510    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:23:05.203706 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:05.203672    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:23:06.819531 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:06.819484    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:23:08.432693 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:08.432652    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:23:08.877920 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:08.877842    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-88a4d9/simulator-w0-88a4d9-kserve-785f84888f-sjbzp"
Nov 03 22:23:08.890553 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:08.890527    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-88a4d9/simulator-w0-88a4d9-kserve-785f84888f-sjbzp"
Nov 03 22:23:09.598505 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:09.598469    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-553753/simulator-w2-553753-kserve-64d77d4dfc-4rlbn"]
Nov 03 22:23:09.601390 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:09.601373    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-553753/simulator-w2-553753-kserve-64d77d4dfc-4rlbn"
Nov 03 22:23:09.635482 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:09.635453    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-553753\"/\"simulator-w2-553753-kserve-self-signed-certs\""
Nov 03 22:23:09.635482 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:09.635492    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-553753\"/\"default-dockercfg-jvvp5\""
Nov 03 22:23:09.637591 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:09.637562    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-553753/simulator-w2-553753-kserve-64d77d4dfc-4rlbn"]
Nov 03 22:23:09.640364 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:09.640346    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-553753\"/\"openshift-service-ca.crt\""
Nov 03 22:23:09.640440 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:09.640408    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-553753\"/\"kube-root-ca.crt\""
Nov 03 22:23:09.705691 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:09.705604    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:23:09.706234 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:09.706190    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e-tmp-dir\") pod \"simulator-w2-553753-kserve-64d77d4dfc-4rlbn\" (UID: \"a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e\") " pod="e2e-models-e2e-worker-w2-553753/simulator-w2-553753-kserve-64d77d4dfc-4rlbn"
Nov 03 22:23:09.706356 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:09.706299    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e-home\") pod \"simulator-w2-553753-kserve-64d77d4dfc-4rlbn\" (UID: \"a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e\") " pod="e2e-models-e2e-worker-w2-553753/simulator-w2-553753-kserve-64d77d4dfc-4rlbn"
Nov 03 22:23:09.706421 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:09.706381    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tflqt\" (UniqueName: \"kubernetes.io/projected/a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e-kube-api-access-tflqt\") pod \"simulator-w2-553753-kserve-64d77d4dfc-4rlbn\" (UID: \"a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e\") " pod="e2e-models-e2e-worker-w2-553753/simulator-w2-553753-kserve-64d77d4dfc-4rlbn"
Nov 03 22:23:09.706480 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:09.706452    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e-tls-certs\") pod \"simulator-w2-553753-kserve-64d77d4dfc-4rlbn\" (UID: \"a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e\") " pod="e2e-models-e2e-worker-w2-553753/simulator-w2-553753-kserve-64d77d4dfc-4rlbn"
Nov 03 22:23:09.706536 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:09.706484    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e-model-cache\") pod \"simulator-w2-553753-kserve-64d77d4dfc-4rlbn\" (UID: \"a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e\") " pod="e2e-models-e2e-worker-w2-553753/simulator-w2-553753-kserve-64d77d4dfc-4rlbn"
Nov 03 22:23:09.706593 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:09.706555    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e-dshm\") pod \"simulator-w2-553753-kserve-64d77d4dfc-4rlbn\" (UID: \"a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e\") " pod="e2e-models-e2e-worker-w2-553753/simulator-w2-553753-kserve-64d77d4dfc-4rlbn"
Nov 03 22:23:09.807715 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:09.807676    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e-tmp-dir\") pod \"simulator-w2-553753-kserve-64d77d4dfc-4rlbn\" (UID: \"a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e\") " pod="e2e-models-e2e-worker-w2-553753/simulator-w2-553753-kserve-64d77d4dfc-4rlbn"
Nov 03 22:23:09.807715 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:09.807713    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e-home\") pod \"simulator-w2-553753-kserve-64d77d4dfc-4rlbn\" (UID: \"a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e\") " pod="e2e-models-e2e-worker-w2-553753/simulator-w2-553753-kserve-64d77d4dfc-4rlbn"
Nov 03 22:23:09.807970 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:09.807761    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tflqt\" (UniqueName: \"kubernetes.io/projected/a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e-kube-api-access-tflqt\") pod \"simulator-w2-553753-kserve-64d77d4dfc-4rlbn\" (UID: \"a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e\") " pod="e2e-models-e2e-worker-w2-553753/simulator-w2-553753-kserve-64d77d4dfc-4rlbn"
Nov 03 22:23:09.807970 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:09.807786    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e-tls-certs\") pod \"simulator-w2-553753-kserve-64d77d4dfc-4rlbn\" (UID: \"a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e\") " pod="e2e-models-e2e-worker-w2-553753/simulator-w2-553753-kserve-64d77d4dfc-4rlbn"
Nov 03 22:23:09.807970 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:09.807816    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e-model-cache\") pod \"simulator-w2-553753-kserve-64d77d4dfc-4rlbn\" (UID: \"a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e\") " pod="e2e-models-e2e-worker-w2-553753/simulator-w2-553753-kserve-64d77d4dfc-4rlbn"
Nov 03 22:23:09.807970 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:09.807910    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e-dshm\") pod \"simulator-w2-553753-kserve-64d77d4dfc-4rlbn\" (UID: \"a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e\") " pod="e2e-models-e2e-worker-w2-553753/simulator-w2-553753-kserve-64d77d4dfc-4rlbn"
Nov 03 22:23:09.808251 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:09.808198    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e-tmp-dir\") pod \"simulator-w2-553753-kserve-64d77d4dfc-4rlbn\" (UID: \"a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e\") " pod="e2e-models-e2e-worker-w2-553753/simulator-w2-553753-kserve-64d77d4dfc-4rlbn"
Nov 03 22:23:09.808331 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:09.808242    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e-home\") pod \"simulator-w2-553753-kserve-64d77d4dfc-4rlbn\" (UID: \"a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e\") " pod="e2e-models-e2e-worker-w2-553753/simulator-w2-553753-kserve-64d77d4dfc-4rlbn"
Nov 03 22:23:09.808444 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:09.808418    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e-model-cache\") pod \"simulator-w2-553753-kserve-64d77d4dfc-4rlbn\" (UID: \"a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e\") " pod="e2e-models-e2e-worker-w2-553753/simulator-w2-553753-kserve-64d77d4dfc-4rlbn"
Nov 03 22:23:09.810871 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:09.810846    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e-dshm\") pod \"simulator-w2-553753-kserve-64d77d4dfc-4rlbn\" (UID: \"a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e\") " pod="e2e-models-e2e-worker-w2-553753/simulator-w2-553753-kserve-64d77d4dfc-4rlbn"
Nov 03 22:23:09.811169 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:09.811150    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e-tls-certs\") pod \"simulator-w2-553753-kserve-64d77d4dfc-4rlbn\" (UID: \"a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e\") " pod="e2e-models-e2e-worker-w2-553753/simulator-w2-553753-kserve-64d77d4dfc-4rlbn"
Nov 03 22:23:09.831707 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:09.831676    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tflqt\" (UniqueName: \"kubernetes.io/projected/a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e-kube-api-access-tflqt\") pod \"simulator-w2-553753-kserve-64d77d4dfc-4rlbn\" (UID: \"a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e\") " pod="e2e-models-e2e-worker-w2-553753/simulator-w2-553753-kserve-64d77d4dfc-4rlbn"
Nov 03 22:23:09.866863 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:09.866779    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-88a4d9/simulator-w0-88a4d9-kserve-785f84888f-sjbzp"
Nov 03 22:23:09.913266 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:09.913199    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-553753/simulator-w2-553753-kserve-64d77d4dfc-4rlbn"
Nov 03 22:23:10.035296 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:10.035261    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:23:10.225930 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:10.225891    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-553753/simulator-w2-553753-kserve-64d77d4dfc-4rlbn"]
Nov 03 22:23:10.229053 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:23:10.229026    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda762e2f1_30fd_4db7_8f5b_67aa1cc6a98e.slice/crio-2f278fb9858b58015c0b65491f3c3d2c6c443c23ee776ca971b60f4b557e4c7f WatchSource:0}: Error finding container 2f278fb9858b58015c0b65491f3c3d2c6c443c23ee776ca971b60f4b557e4c7f: Status 404 returned error can't find the container with id 2f278fb9858b58015c0b65491f3c3d2c6c443c23ee776ca971b60f4b557e4c7f
Nov 03 22:23:10.860660 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:10.860622    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-553753/simulator-w2-553753-kserve-64d77d4dfc-4rlbn" event={"ID":"a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e","Type":"ContainerStarted","Data":"4b94a3ff05734cad306a301f260f406dd2548c0943f2b6b894ce910563dbbb81"}
Nov 03 22:23:10.860660 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:10.860663    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-553753/simulator-w2-553753-kserve-64d77d4dfc-4rlbn" event={"ID":"a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e","Type":"ContainerStarted","Data":"2f278fb9858b58015c0b65491f3c3d2c6c443c23ee776ca971b60f4b557e4c7f"}
Nov 03 22:23:10.902362 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:10.902308    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-553753/simulator-w2-553753-kserve-64d77d4dfc-4rlbn" podStartSLOduration=1.902290756 podStartE2EDuration="1.902290756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:23:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:23:10.901644784 +0000 UTC m=+614.927500647" watchObservedRunningTime="2026-11-03 22:23:10.902290756 +0000 UTC m=+614.928146617"
Nov 03 22:23:10.913789 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:10.913759    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-553753/simulator-w2-553753-kserve-64d77d4dfc-4rlbn"
Nov 03 22:23:11.618695 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:11.618658    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:23:13.434230 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:13.434191    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:23:15.122633 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:15.122599    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:23:16.640491 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:16.640449    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:23:18.259040 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:18.258992    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:23:19.898819 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:19.898782    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:23:19.913517 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:19.913486    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-553753/simulator-w2-553753-kserve-64d77d4dfc-4rlbn"
Nov 03 22:23:19.927869 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:19.927841    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-553753/simulator-w2-553753-kserve-64d77d4dfc-4rlbn"
Nov 03 22:23:20.923440 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:20.923407    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-553753/simulator-w2-553753-kserve-64d77d4dfc-4rlbn"
Nov 03 22:23:20.980112 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:20.980050    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:23:21.494396 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:21.494345    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:23:23.103586 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:23.103553    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:23:24.714578 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:24.714544    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:23:26.277120 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:26.277076    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:23:27.890143 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:27.890102    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:23:29.320347 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:29.320313    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-5187e1/premium-w3-5187e1-kserve-56bf8788d4-pp8st"]
Nov 03 22:23:29.325149 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:29.325125    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5187e1/premium-w3-5187e1-kserve-56bf8788d4-pp8st"
Nov 03 22:23:29.353171 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:29.353140    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-5187e1\"/\"default-dockercfg-5nm2d\""
Nov 03 22:23:29.353342 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:29.353197    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-5187e1\"/\"openshift-service-ca.crt\""
Nov 03 22:23:29.353342 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:29.353255    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-5187e1\"/\"kube-root-ca.crt\""
Nov 03 22:23:29.355306 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:29.355286    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-5187e1/premium-w3-5187e1-kserve-56bf8788d4-pp8st"]
Nov 03 22:23:29.405859 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:29.405822    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a70ecfe9-020a-4331-b67c-aaaa46ed1eb1-tmp-dir\") pod \"premium-w3-5187e1-kserve-56bf8788d4-pp8st\" (UID: \"a70ecfe9-020a-4331-b67c-aaaa46ed1eb1\") " pod="e2e-models-e2e-worker-w3-5187e1/premium-w3-5187e1-kserve-56bf8788d4-pp8st"
Nov 03 22:23:29.405859 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:29.405862    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a70ecfe9-020a-4331-b67c-aaaa46ed1eb1-model-cache\") pod \"premium-w3-5187e1-kserve-56bf8788d4-pp8st\" (UID: \"a70ecfe9-020a-4331-b67c-aaaa46ed1eb1\") " pod="e2e-models-e2e-worker-w3-5187e1/premium-w3-5187e1-kserve-56bf8788d4-pp8st"
Nov 03 22:23:29.406122 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:29.405897    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsfr4\" (UniqueName: \"kubernetes.io/projected/a70ecfe9-020a-4331-b67c-aaaa46ed1eb1-kube-api-access-xsfr4\") pod \"premium-w3-5187e1-kserve-56bf8788d4-pp8st\" (UID: \"a70ecfe9-020a-4331-b67c-aaaa46ed1eb1\") " pod="e2e-models-e2e-worker-w3-5187e1/premium-w3-5187e1-kserve-56bf8788d4-pp8st"
Nov 03 22:23:29.406122 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:29.406021    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a70ecfe9-020a-4331-b67c-aaaa46ed1eb1-tls-certs\") pod \"premium-w3-5187e1-kserve-56bf8788d4-pp8st\" (UID: \"a70ecfe9-020a-4331-b67c-aaaa46ed1eb1\") " pod="e2e-models-e2e-worker-w3-5187e1/premium-w3-5187e1-kserve-56bf8788d4-pp8st"
Nov 03 22:23:29.406122 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:29.406071    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a70ecfe9-020a-4331-b67c-aaaa46ed1eb1-home\") pod \"premium-w3-5187e1-kserve-56bf8788d4-pp8st\" (UID: \"a70ecfe9-020a-4331-b67c-aaaa46ed1eb1\") " pod="e2e-models-e2e-worker-w3-5187e1/premium-w3-5187e1-kserve-56bf8788d4-pp8st"
Nov 03 22:23:29.406339 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:29.406179    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a70ecfe9-020a-4331-b67c-aaaa46ed1eb1-dshm\") pod \"premium-w3-5187e1-kserve-56bf8788d4-pp8st\" (UID: \"a70ecfe9-020a-4331-b67c-aaaa46ed1eb1\") " pod="e2e-models-e2e-worker-w3-5187e1/premium-w3-5187e1-kserve-56bf8788d4-pp8st"
Nov 03 22:23:29.408078 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:29.408047    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:23:29.441825 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:29.441791    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-5187e1\"/\"premium-w3-5187e1-kserve-self-signed-certs\""
Nov 03 22:23:29.507770 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:29.507733    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a70ecfe9-020a-4331-b67c-aaaa46ed1eb1-tmp-dir\") pod \"premium-w3-5187e1-kserve-56bf8788d4-pp8st\" (UID: \"a70ecfe9-020a-4331-b67c-aaaa46ed1eb1\") " pod="e2e-models-e2e-worker-w3-5187e1/premium-w3-5187e1-kserve-56bf8788d4-pp8st"
Nov 03 22:23:29.508031 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:29.507995    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a70ecfe9-020a-4331-b67c-aaaa46ed1eb1-model-cache\") pod \"premium-w3-5187e1-kserve-56bf8788d4-pp8st\" (UID: \"a70ecfe9-020a-4331-b67c-aaaa46ed1eb1\") " pod="e2e-models-e2e-worker-w3-5187e1/premium-w3-5187e1-kserve-56bf8788d4-pp8st"
Nov 03 22:23:29.508182 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:29.508069    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsfr4\" (UniqueName: \"kubernetes.io/projected/a70ecfe9-020a-4331-b67c-aaaa46ed1eb1-kube-api-access-xsfr4\") pod \"premium-w3-5187e1-kserve-56bf8788d4-pp8st\" (UID: \"a70ecfe9-020a-4331-b67c-aaaa46ed1eb1\") " pod="e2e-models-e2e-worker-w3-5187e1/premium-w3-5187e1-kserve-56bf8788d4-pp8st"
Nov 03 22:23:29.508182 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:29.508130    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a70ecfe9-020a-4331-b67c-aaaa46ed1eb1-tls-certs\") pod \"premium-w3-5187e1-kserve-56bf8788d4-pp8st\" (UID: \"a70ecfe9-020a-4331-b67c-aaaa46ed1eb1\") " pod="e2e-models-e2e-worker-w3-5187e1/premium-w3-5187e1-kserve-56bf8788d4-pp8st"
Nov 03 22:23:29.508344 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:29.508181    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a70ecfe9-020a-4331-b67c-aaaa46ed1eb1-home\") pod \"premium-w3-5187e1-kserve-56bf8788d4-pp8st\" (UID: \"a70ecfe9-020a-4331-b67c-aaaa46ed1eb1\") " pod="e2e-models-e2e-worker-w3-5187e1/premium-w3-5187e1-kserve-56bf8788d4-pp8st"
Nov 03 22:23:29.508344 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:29.508281    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a70ecfe9-020a-4331-b67c-aaaa46ed1eb1-dshm\") pod \"premium-w3-5187e1-kserve-56bf8788d4-pp8st\" (UID: \"a70ecfe9-020a-4331-b67c-aaaa46ed1eb1\") " pod="e2e-models-e2e-worker-w3-5187e1/premium-w3-5187e1-kserve-56bf8788d4-pp8st"
Nov 03 22:23:29.509059 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:29.509033    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a70ecfe9-020a-4331-b67c-aaaa46ed1eb1-tmp-dir\") pod \"premium-w3-5187e1-kserve-56bf8788d4-pp8st\" (UID: \"a70ecfe9-020a-4331-b67c-aaaa46ed1eb1\") " pod="e2e-models-e2e-worker-w3-5187e1/premium-w3-5187e1-kserve-56bf8788d4-pp8st"
Nov 03 22:23:29.509257 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:29.509184    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a70ecfe9-020a-4331-b67c-aaaa46ed1eb1-home\") pod \"premium-w3-5187e1-kserve-56bf8788d4-pp8st\" (UID: \"a70ecfe9-020a-4331-b67c-aaaa46ed1eb1\") " pod="e2e-models-e2e-worker-w3-5187e1/premium-w3-5187e1-kserve-56bf8788d4-pp8st"
Nov 03 22:23:29.509363 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:29.509259    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a70ecfe9-020a-4331-b67c-aaaa46ed1eb1-model-cache\") pod \"premium-w3-5187e1-kserve-56bf8788d4-pp8st\" (UID: \"a70ecfe9-020a-4331-b67c-aaaa46ed1eb1\") " pod="e2e-models-e2e-worker-w3-5187e1/premium-w3-5187e1-kserve-56bf8788d4-pp8st"
Nov 03 22:23:29.511423 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:29.511396    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a70ecfe9-020a-4331-b67c-aaaa46ed1eb1-dshm\") pod \"premium-w3-5187e1-kserve-56bf8788d4-pp8st\" (UID: \"a70ecfe9-020a-4331-b67c-aaaa46ed1eb1\") " pod="e2e-models-e2e-worker-w3-5187e1/premium-w3-5187e1-kserve-56bf8788d4-pp8st"
Nov 03 22:23:29.517811 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:29.517784    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a70ecfe9-020a-4331-b67c-aaaa46ed1eb1-tls-certs\") pod \"premium-w3-5187e1-kserve-56bf8788d4-pp8st\" (UID: \"a70ecfe9-020a-4331-b67c-aaaa46ed1eb1\") " pod="e2e-models-e2e-worker-w3-5187e1/premium-w3-5187e1-kserve-56bf8788d4-pp8st"
Nov 03 22:23:29.532797 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:29.532768    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsfr4\" (UniqueName: \"kubernetes.io/projected/a70ecfe9-020a-4331-b67c-aaaa46ed1eb1-kube-api-access-xsfr4\") pod \"premium-w3-5187e1-kserve-56bf8788d4-pp8st\" (UID: \"a70ecfe9-020a-4331-b67c-aaaa46ed1eb1\") " pod="e2e-models-e2e-worker-w3-5187e1/premium-w3-5187e1-kserve-56bf8788d4-pp8st"
Nov 03 22:23:29.611168 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:29.611091    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:23:29.637756 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:29.637721    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5187e1/premium-w3-5187e1-kserve-56bf8788d4-pp8st"
Nov 03 22:23:29.939784 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:29.939761    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-5187e1/premium-w3-5187e1-kserve-56bf8788d4-pp8st"]
Nov 03 22:23:29.941915 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:23:29.941884    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda70ecfe9_020a_4331_b67c_aaaa46ed1eb1.slice/crio-ed2c6fd811d390d3c50d30f9e52f1c8d2ffcf20dab099c2a4422d835765328dd WatchSource:0}: Error finding container ed2c6fd811d390d3c50d30f9e52f1c8d2ffcf20dab099c2a4422d835765328dd: Status 404 returned error can't find the container with id ed2c6fd811d390d3c50d30f9e52f1c8d2ffcf20dab099c2a4422d835765328dd
Nov 03 22:23:29.962879 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:29.962848    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5187e1/premium-w3-5187e1-kserve-56bf8788d4-pp8st" event={"ID":"a70ecfe9-020a-4331-b67c-aaaa46ed1eb1","Type":"ContainerStarted","Data":"ed2c6fd811d390d3c50d30f9e52f1c8d2ffcf20dab099c2a4422d835765328dd"}
Nov 03 22:23:30.968972 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:30.968934    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5187e1/premium-w3-5187e1-kserve-56bf8788d4-pp8st" event={"ID":"a70ecfe9-020a-4331-b67c-aaaa46ed1eb1","Type":"ContainerStarted","Data":"802fa2f1254e8db4a40a463b19e2778d32d1dcfa4dbc494eeb32c6872b05cede"}
Nov 03 22:23:31.010104 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:31.010047    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-5187e1/premium-w3-5187e1-kserve-56bf8788d4-pp8st" podStartSLOduration=2.010033156 podStartE2EDuration="2.010033156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:23:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:23:31.009716911 +0000 UTC m=+635.035572784" watchObservedRunningTime="2026-11-03 22:23:31.010033156 +0000 UTC m=+635.035889019"
Nov 03 22:23:31.198137 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:31.198107    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:23:31.259346 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:23:31.259265    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:23:31.638037 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:31.637961    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5187e1/premium-w3-5187e1-kserve-56bf8788d4-pp8st"
Nov 03 22:23:32.792983 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:32.792948    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:23:33.976052 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:33.976013    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:23:34.593236 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:34.593187    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:23:35.825086 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:35.825047    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:23:35.980624 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:35.980586    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:23:36.131881 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:36.131783    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:23:37.692344 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:37.692314    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:23:39.325248 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:39.325180    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:23:39.638104 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:39.638022    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-5187e1/premium-w3-5187e1-kserve-56bf8788d4-pp8st"
Nov 03 22:23:39.650163 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:39.650137    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-5187e1/premium-w3-5187e1-kserve-56bf8788d4-pp8st"
Nov 03 22:23:39.868299 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:39.868268    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:23:40.030124 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:40.030091    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5187e1/premium-w3-5187e1-kserve-56bf8788d4-pp8st"
Nov 03 22:23:40.828428 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:40.828393    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:23:42.409560 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:42.409526    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:23:42.928824 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:42.928683    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-88a4d9/premium-w0-88a4d9-kserve-5559d75b7b-vqllq"]
Nov 03 22:23:42.932460 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:42.932439    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-88a4d9/premium-w0-88a4d9-kserve-5559d75b7b-vqllq"
Nov 03 22:23:42.960198 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:42.960167    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-88a4d9\"/\"premium-w0-88a4d9-kserve-self-signed-certs\""
Nov 03 22:23:42.962247 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:42.962206    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-88a4d9/premium-w0-88a4d9-kserve-5559d75b7b-vqllq"]
Nov 03 22:23:43.019846 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:43.019797    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:23:43.050377 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:43.050343    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05206737-4ca3-41fc-b35a-37c625cc182c-dshm\") pod \"premium-w0-88a4d9-kserve-5559d75b7b-vqllq\" (UID: \"05206737-4ca3-41fc-b35a-37c625cc182c\") " pod="e2e-models-e2e-worker-w0-88a4d9/premium-w0-88a4d9-kserve-5559d75b7b-vqllq"
Nov 03 22:23:43.050540 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:43.050390    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05206737-4ca3-41fc-b35a-37c625cc182c-home\") pod \"premium-w0-88a4d9-kserve-5559d75b7b-vqllq\" (UID: \"05206737-4ca3-41fc-b35a-37c625cc182c\") " pod="e2e-models-e2e-worker-w0-88a4d9/premium-w0-88a4d9-kserve-5559d75b7b-vqllq"
Nov 03 22:23:43.050540 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:43.050480    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thvpg\" (UniqueName: \"kubernetes.io/projected/05206737-4ca3-41fc-b35a-37c625cc182c-kube-api-access-thvpg\") pod \"premium-w0-88a4d9-kserve-5559d75b7b-vqllq\" (UID: \"05206737-4ca3-41fc-b35a-37c625cc182c\") " pod="e2e-models-e2e-worker-w0-88a4d9/premium-w0-88a4d9-kserve-5559d75b7b-vqllq"
Nov 03 22:23:43.050631 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:43.050607    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05206737-4ca3-41fc-b35a-37c625cc182c-tmp-dir\") pod \"premium-w0-88a4d9-kserve-5559d75b7b-vqllq\" (UID: \"05206737-4ca3-41fc-b35a-37c625cc182c\") " pod="e2e-models-e2e-worker-w0-88a4d9/premium-w0-88a4d9-kserve-5559d75b7b-vqllq"
Nov 03 22:23:43.050671 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:43.050655    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05206737-4ca3-41fc-b35a-37c625cc182c-tls-certs\") pod \"premium-w0-88a4d9-kserve-5559d75b7b-vqllq\" (UID: \"05206737-4ca3-41fc-b35a-37c625cc182c\") " pod="e2e-models-e2e-worker-w0-88a4d9/premium-w0-88a4d9-kserve-5559d75b7b-vqllq"
Nov 03 22:23:43.050799 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:43.050783    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05206737-4ca3-41fc-b35a-37c625cc182c-model-cache\") pod \"premium-w0-88a4d9-kserve-5559d75b7b-vqllq\" (UID: \"05206737-4ca3-41fc-b35a-37c625cc182c\") " pod="e2e-models-e2e-worker-w0-88a4d9/premium-w0-88a4d9-kserve-5559d75b7b-vqllq"
Nov 03 22:23:43.151899 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:43.151856    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05206737-4ca3-41fc-b35a-37c625cc182c-model-cache\") pod \"premium-w0-88a4d9-kserve-5559d75b7b-vqllq\" (UID: \"05206737-4ca3-41fc-b35a-37c625cc182c\") " pod="e2e-models-e2e-worker-w0-88a4d9/premium-w0-88a4d9-kserve-5559d75b7b-vqllq"
Nov 03 22:23:43.152100 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:43.152036    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05206737-4ca3-41fc-b35a-37c625cc182c-dshm\") pod \"premium-w0-88a4d9-kserve-5559d75b7b-vqllq\" (UID: \"05206737-4ca3-41fc-b35a-37c625cc182c\") " pod="e2e-models-e2e-worker-w0-88a4d9/premium-w0-88a4d9-kserve-5559d75b7b-vqllq"
Nov 03 22:23:43.152100 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:43.152085    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05206737-4ca3-41fc-b35a-37c625cc182c-home\") pod \"premium-w0-88a4d9-kserve-5559d75b7b-vqllq\" (UID: \"05206737-4ca3-41fc-b35a-37c625cc182c\") " pod="e2e-models-e2e-worker-w0-88a4d9/premium-w0-88a4d9-kserve-5559d75b7b-vqllq"
Nov 03 22:23:43.152242 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:43.152156    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thvpg\" (UniqueName: \"kubernetes.io/projected/05206737-4ca3-41fc-b35a-37c625cc182c-kube-api-access-thvpg\") pod \"premium-w0-88a4d9-kserve-5559d75b7b-vqllq\" (UID: \"05206737-4ca3-41fc-b35a-37c625cc182c\") " pod="e2e-models-e2e-worker-w0-88a4d9/premium-w0-88a4d9-kserve-5559d75b7b-vqllq"
Nov 03 22:23:43.152305 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:43.152273    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05206737-4ca3-41fc-b35a-37c625cc182c-tmp-dir\") pod \"premium-w0-88a4d9-kserve-5559d75b7b-vqllq\" (UID: \"05206737-4ca3-41fc-b35a-37c625cc182c\") " pod="e2e-models-e2e-worker-w0-88a4d9/premium-w0-88a4d9-kserve-5559d75b7b-vqllq"
Nov 03 22:23:43.152368 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:43.152308    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05206737-4ca3-41fc-b35a-37c625cc182c-model-cache\") pod \"premium-w0-88a4d9-kserve-5559d75b7b-vqllq\" (UID: \"05206737-4ca3-41fc-b35a-37c625cc182c\") " pod="e2e-models-e2e-worker-w0-88a4d9/premium-w0-88a4d9-kserve-5559d75b7b-vqllq"
Nov 03 22:23:43.152368 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:43.152330    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05206737-4ca3-41fc-b35a-37c625cc182c-tls-certs\") pod \"premium-w0-88a4d9-kserve-5559d75b7b-vqllq\" (UID: \"05206737-4ca3-41fc-b35a-37c625cc182c\") " pod="e2e-models-e2e-worker-w0-88a4d9/premium-w0-88a4d9-kserve-5559d75b7b-vqllq"
Nov 03 22:23:43.152477 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:43.152399    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05206737-4ca3-41fc-b35a-37c625cc182c-home\") pod \"premium-w0-88a4d9-kserve-5559d75b7b-vqllq\" (UID: \"05206737-4ca3-41fc-b35a-37c625cc182c\") " pod="e2e-models-e2e-worker-w0-88a4d9/premium-w0-88a4d9-kserve-5559d75b7b-vqllq"
Nov 03 22:23:43.152575 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:43.152553    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05206737-4ca3-41fc-b35a-37c625cc182c-tmp-dir\") pod \"premium-w0-88a4d9-kserve-5559d75b7b-vqllq\" (UID: \"05206737-4ca3-41fc-b35a-37c625cc182c\") " pod="e2e-models-e2e-worker-w0-88a4d9/premium-w0-88a4d9-kserve-5559d75b7b-vqllq"
Nov 03 22:23:43.154538 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:43.154509    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05206737-4ca3-41fc-b35a-37c625cc182c-dshm\") pod \"premium-w0-88a4d9-kserve-5559d75b7b-vqllq\" (UID: \"05206737-4ca3-41fc-b35a-37c625cc182c\") " pod="e2e-models-e2e-worker-w0-88a4d9/premium-w0-88a4d9-kserve-5559d75b7b-vqllq"
Nov 03 22:23:43.154691 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:43.154672    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05206737-4ca3-41fc-b35a-37c625cc182c-tls-certs\") pod \"premium-w0-88a4d9-kserve-5559d75b7b-vqllq\" (UID: \"05206737-4ca3-41fc-b35a-37c625cc182c\") " pod="e2e-models-e2e-worker-w0-88a4d9/premium-w0-88a4d9-kserve-5559d75b7b-vqllq"
Nov 03 22:23:43.175092 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:43.175061    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thvpg\" (UniqueName: \"kubernetes.io/projected/05206737-4ca3-41fc-b35a-37c625cc182c-kube-api-access-thvpg\") pod \"premium-w0-88a4d9-kserve-5559d75b7b-vqllq\" (UID: \"05206737-4ca3-41fc-b35a-37c625cc182c\") " pod="e2e-models-e2e-worker-w0-88a4d9/premium-w0-88a4d9-kserve-5559d75b7b-vqllq"
Nov 03 22:23:43.246181 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:43.246100    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-88a4d9/premium-w0-88a4d9-kserve-5559d75b7b-vqllq"
Nov 03 22:23:43.575429 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:43.575403    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-88a4d9/premium-w0-88a4d9-kserve-5559d75b7b-vqllq"]
Nov 03 22:23:43.577124 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:23:43.577101    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod05206737_4ca3_41fc_b35a_37c625cc182c.slice/crio-76dc9f00e4974208cb67becda0970aa924d111b0553f24e694a25bbc03b28193 WatchSource:0}: Error finding container 76dc9f00e4974208cb67becda0970aa924d111b0553f24e694a25bbc03b28193: Status 404 returned error can't find the container with id 76dc9f00e4974208cb67becda0970aa924d111b0553f24e694a25bbc03b28193
Nov 03 22:23:44.032638 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:44.032603    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-88a4d9/premium-w0-88a4d9-kserve-5559d75b7b-vqllq" event={"ID":"05206737-4ca3-41fc-b35a-37c625cc182c","Type":"ContainerStarted","Data":"d71101cf7f8cac4b13219889ee6bdde6099376f3a2df441098177a241ab2e114"}
Nov 03 22:23:44.032831 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:44.032645    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-88a4d9/premium-w0-88a4d9-kserve-5559d75b7b-vqllq" event={"ID":"05206737-4ca3-41fc-b35a-37c625cc182c","Type":"ContainerStarted","Data":"76dc9f00e4974208cb67becda0970aa924d111b0553f24e694a25bbc03b28193"}
Nov 03 22:23:44.088817 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:44.088763    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-88a4d9/premium-w0-88a4d9-kserve-5559d75b7b-vqllq" podStartSLOduration=2.088750616 podStartE2EDuration="2.088750616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:23:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:23:44.087628273 +0000 UTC m=+648.113484135" watchObservedRunningTime="2026-11-03 22:23:44.088750616 +0000 UTC m=+648.114606477"
Nov 03 22:23:44.111550 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:44.111512    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:23:44.247111 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:44.247074    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-88a4d9/premium-w0-88a4d9-kserve-5559d75b7b-vqllq"
Nov 03 22:23:45.215962 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:45.215924    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:23:45.411084 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:45.411050    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:23:45.952695 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:45.952630    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:23:46.653764 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:46.653726    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:23:47.612545 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:47.612506    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:23:49.206987 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:49.206951    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:23:50.721432 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:50.721387    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:23:51.839113 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:51.839080    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-50d738/premium-w4-50d738-kserve-59ccd7f995-xbfq2"]
Nov 03 22:23:51.842943 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:51.842923    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-50d738/premium-w4-50d738-kserve-59ccd7f995-xbfq2"
Nov 03 22:23:51.859801 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:51.859776    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-50d738\"/\"kube-root-ca.crt\""
Nov 03 22:23:51.860029 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:51.860013    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-50d738\"/\"premium-w4-50d738-kserve-self-signed-certs\""
Nov 03 22:23:51.872332 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:51.872308    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-50d738\"/\"default-dockercfg-rndwk\""
Nov 03 22:23:51.874449 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:51.874425    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-50d738/premium-w4-50d738-kserve-59ccd7f995-xbfq2"]
Nov 03 22:23:51.914145 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:51.914109    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:23:51.949823 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:51.949777    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa0a680b-8f0d-4b19-8217-82282764511a-model-cache\") pod \"premium-w4-50d738-kserve-59ccd7f995-xbfq2\" (UID: \"fa0a680b-8f0d-4b19-8217-82282764511a\") " pod="e2e-models-e2e-worker-w4-50d738/premium-w4-50d738-kserve-59ccd7f995-xbfq2"
Nov 03 22:23:51.949980 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:51.949844    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa0a680b-8f0d-4b19-8217-82282764511a-home\") pod \"premium-w4-50d738-kserve-59ccd7f995-xbfq2\" (UID: \"fa0a680b-8f0d-4b19-8217-82282764511a\") " pod="e2e-models-e2e-worker-w4-50d738/premium-w4-50d738-kserve-59ccd7f995-xbfq2"
Nov 03 22:23:51.950041 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:51.949972    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8b5h\" (UniqueName: \"kubernetes.io/projected/fa0a680b-8f0d-4b19-8217-82282764511a-kube-api-access-h8b5h\") pod \"premium-w4-50d738-kserve-59ccd7f995-xbfq2\" (UID: \"fa0a680b-8f0d-4b19-8217-82282764511a\") " pod="e2e-models-e2e-worker-w4-50d738/premium-w4-50d738-kserve-59ccd7f995-xbfq2"
Nov 03 22:23:51.950041 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:51.950011    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa0a680b-8f0d-4b19-8217-82282764511a-tmp-dir\") pod \"premium-w4-50d738-kserve-59ccd7f995-xbfq2\" (UID: \"fa0a680b-8f0d-4b19-8217-82282764511a\") " pod="e2e-models-e2e-worker-w4-50d738/premium-w4-50d738-kserve-59ccd7f995-xbfq2"
Nov 03 22:23:51.950041 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:51.950038    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa0a680b-8f0d-4b19-8217-82282764511a-tls-certs\") pod \"premium-w4-50d738-kserve-59ccd7f995-xbfq2\" (UID: \"fa0a680b-8f0d-4b19-8217-82282764511a\") " pod="e2e-models-e2e-worker-w4-50d738/premium-w4-50d738-kserve-59ccd7f995-xbfq2"
Nov 03 22:23:51.950198 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:51.950175    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa0a680b-8f0d-4b19-8217-82282764511a-dshm\") pod \"premium-w4-50d738-kserve-59ccd7f995-xbfq2\" (UID: \"fa0a680b-8f0d-4b19-8217-82282764511a\") " pod="e2e-models-e2e-worker-w4-50d738/premium-w4-50d738-kserve-59ccd7f995-xbfq2"
Nov 03 22:23:51.959973 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:51.959955    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-50d738\"/\"openshift-service-ca.crt\""
Nov 03 22:23:52.051613 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:52.051572    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8b5h\" (UniqueName: \"kubernetes.io/projected/fa0a680b-8f0d-4b19-8217-82282764511a-kube-api-access-h8b5h\") pod \"premium-w4-50d738-kserve-59ccd7f995-xbfq2\" (UID: \"fa0a680b-8f0d-4b19-8217-82282764511a\") " pod="e2e-models-e2e-worker-w4-50d738/premium-w4-50d738-kserve-59ccd7f995-xbfq2"
Nov 03 22:23:52.051613 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:52.051618    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa0a680b-8f0d-4b19-8217-82282764511a-tmp-dir\") pod \"premium-w4-50d738-kserve-59ccd7f995-xbfq2\" (UID: \"fa0a680b-8f0d-4b19-8217-82282764511a\") " pod="e2e-models-e2e-worker-w4-50d738/premium-w4-50d738-kserve-59ccd7f995-xbfq2"
Nov 03 22:23:52.051889 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:52.051641    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa0a680b-8f0d-4b19-8217-82282764511a-tls-certs\") pod \"premium-w4-50d738-kserve-59ccd7f995-xbfq2\" (UID: \"fa0a680b-8f0d-4b19-8217-82282764511a\") " pod="e2e-models-e2e-worker-w4-50d738/premium-w4-50d738-kserve-59ccd7f995-xbfq2"
Nov 03 22:23:52.051889 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:52.051696    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa0a680b-8f0d-4b19-8217-82282764511a-dshm\") pod \"premium-w4-50d738-kserve-59ccd7f995-xbfq2\" (UID: \"fa0a680b-8f0d-4b19-8217-82282764511a\") " pod="e2e-models-e2e-worker-w4-50d738/premium-w4-50d738-kserve-59ccd7f995-xbfq2"
Nov 03 22:23:52.051889 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:52.051804    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa0a680b-8f0d-4b19-8217-82282764511a-model-cache\") pod \"premium-w4-50d738-kserve-59ccd7f995-xbfq2\" (UID: \"fa0a680b-8f0d-4b19-8217-82282764511a\") " pod="e2e-models-e2e-worker-w4-50d738/premium-w4-50d738-kserve-59ccd7f995-xbfq2"
Nov 03 22:23:52.051889 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:52.051860    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa0a680b-8f0d-4b19-8217-82282764511a-home\") pod \"premium-w4-50d738-kserve-59ccd7f995-xbfq2\" (UID: \"fa0a680b-8f0d-4b19-8217-82282764511a\") " pod="e2e-models-e2e-worker-w4-50d738/premium-w4-50d738-kserve-59ccd7f995-xbfq2"
Nov 03 22:23:52.052125 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:52.052099    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa0a680b-8f0d-4b19-8217-82282764511a-tmp-dir\") pod \"premium-w4-50d738-kserve-59ccd7f995-xbfq2\" (UID: \"fa0a680b-8f0d-4b19-8217-82282764511a\") " pod="e2e-models-e2e-worker-w4-50d738/premium-w4-50d738-kserve-59ccd7f995-xbfq2"
Nov 03 22:23:52.052206 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:52.052184    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa0a680b-8f0d-4b19-8217-82282764511a-home\") pod \"premium-w4-50d738-kserve-59ccd7f995-xbfq2\" (UID: \"fa0a680b-8f0d-4b19-8217-82282764511a\") " pod="e2e-models-e2e-worker-w4-50d738/premium-w4-50d738-kserve-59ccd7f995-xbfq2"
Nov 03 22:23:52.052627 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:52.052591    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa0a680b-8f0d-4b19-8217-82282764511a-model-cache\") pod \"premium-w4-50d738-kserve-59ccd7f995-xbfq2\" (UID: \"fa0a680b-8f0d-4b19-8217-82282764511a\") " pod="e2e-models-e2e-worker-w4-50d738/premium-w4-50d738-kserve-59ccd7f995-xbfq2"
Nov 03 22:23:52.055146 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:52.055123    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa0a680b-8f0d-4b19-8217-82282764511a-tls-certs\") pod \"premium-w4-50d738-kserve-59ccd7f995-xbfq2\" (UID: \"fa0a680b-8f0d-4b19-8217-82282764511a\") " pod="e2e-models-e2e-worker-w4-50d738/premium-w4-50d738-kserve-59ccd7f995-xbfq2"
Nov 03 22:23:52.062417 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:52.062392    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa0a680b-8f0d-4b19-8217-82282764511a-dshm\") pod \"premium-w4-50d738-kserve-59ccd7f995-xbfq2\" (UID: \"fa0a680b-8f0d-4b19-8217-82282764511a\") " pod="e2e-models-e2e-worker-w4-50d738/premium-w4-50d738-kserve-59ccd7f995-xbfq2"
Nov 03 22:23:52.075691 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:52.075661    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8b5h\" (UniqueName: \"kubernetes.io/projected/fa0a680b-8f0d-4b19-8217-82282764511a-kube-api-access-h8b5h\") pod \"premium-w4-50d738-kserve-59ccd7f995-xbfq2\" (UID: \"fa0a680b-8f0d-4b19-8217-82282764511a\") " pod="e2e-models-e2e-worker-w4-50d738/premium-w4-50d738-kserve-59ccd7f995-xbfq2"
Nov 03 22:23:52.156059 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:52.155968    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-50d738/premium-w4-50d738-kserve-59ccd7f995-xbfq2"
Nov 03 22:23:52.229120 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:52.229091    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:23:52.454081 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:52.454057    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-50d738/premium-w4-50d738-kserve-59ccd7f995-xbfq2"]
Nov 03 22:23:52.456432 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:23:52.456404    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfa0a680b_8f0d_4b19_8217_82282764511a.slice/crio-56bb35624f160f6135b2dc0ae54751847a5eb8d14aac5ab97690df02e2a1294b WatchSource:0}: Error finding container 56bb35624f160f6135b2dc0ae54751847a5eb8d14aac5ab97690df02e2a1294b: Status 404 returned error can't find the container with id 56bb35624f160f6135b2dc0ae54751847a5eb8d14aac5ab97690df02e2a1294b
Nov 03 22:23:53.081565 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:53.081535    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-50d738/premium-w4-50d738-kserve-59ccd7f995-xbfq2" event={"ID":"fa0a680b-8f0d-4b19-8217-82282764511a","Type":"ContainerStarted","Data":"6463c2aea4341ac22e57b41fdde2032b0ff43ec2db6c342ed825e2a7e11cabf1"}
Nov 03 22:23:53.081565 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:53.081569    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-50d738/premium-w4-50d738-kserve-59ccd7f995-xbfq2" event={"ID":"fa0a680b-8f0d-4b19-8217-82282764511a","Type":"ContainerStarted","Data":"56bb35624f160f6135b2dc0ae54751847a5eb8d14aac5ab97690df02e2a1294b"}
Nov 03 22:23:53.122950 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:53.122901    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-50d738/premium-w4-50d738-kserve-59ccd7f995-xbfq2" podStartSLOduration=2.122888201 podStartE2EDuration="2.122888201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:23:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:23:53.122254164 +0000 UTC m=+657.148110026" watchObservedRunningTime="2026-11-03 22:23:53.122888201 +0000 UTC m=+657.148744106"
Nov 03 22:23:53.156926 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:53.156894    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-50d738/premium-w4-50d738-kserve-59ccd7f995-xbfq2"
Nov 03 22:23:53.247147 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:53.247109    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-88a4d9/premium-w0-88a4d9-kserve-5559d75b7b-vqllq"
Nov 03 22:23:53.260435 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:53.260406    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-88a4d9/premium-w0-88a4d9-kserve-5559d75b7b-vqllq"
Nov 03 22:23:53.823477 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:53.823438    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:23:54.104130 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:54.104045    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-88a4d9/premium-w0-88a4d9-kserve-5559d75b7b-vqllq"
Nov 03 22:23:55.624831 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:55.624790    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:23:55.779037 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:55.778997    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:23:57.321115 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:57.321082    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:23:59.015855 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:23:59.015823    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:24:00.625952 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:00.625896    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:24:01.258694 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:24:01.258651    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:24:02.156385 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:02.156353    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-50d738/premium-w4-50d738-kserve-59ccd7f995-xbfq2"
Nov 03 22:24:02.169037 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:02.169015    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-50d738/premium-w4-50d738-kserve-59ccd7f995-xbfq2"
Nov 03 22:24:02.270205 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:02.270135    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:24:02.850748 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:02.850709    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:24:02.996866 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:02.996813    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:24:03.149503 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:03.149422    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-50d738/premium-w4-50d738-kserve-59ccd7f995-xbfq2"
Nov 03 22:24:03.798904 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:03.798870    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:24:05.494688 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:05.494656    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:24:07.294938 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:07.294880    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:24:08.905064 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:08.905028    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:24:10.402892 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:10.402814    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:24:11.907388 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:11.907351    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:24:13.307501 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:13.307464    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b"]
Nov 03 22:24:13.311048 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:13.311028    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b"
Nov 03 22:24:13.326196 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:13.326170    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-88a4d9-openshift-default-dockercfg-ph54l\""
Nov 03 22:24:13.340662 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:13.340626    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b"]
Nov 03 22:24:13.377496 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:13.377449    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3553842-86f5-4f08-bc2f-4e7c83e4ca0e-istiod-ca-cert\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b\" (UID: \"c3553842-86f5-4f08-bc2f-4e7c83e4ca0e\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b"
Nov 03 22:24:13.377678 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:13.377602    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3553842-86f5-4f08-bc2f-4e7c83e4ca0e-workload-socket\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b\" (UID: \"c3553842-86f5-4f08-bc2f-4e7c83e4ca0e\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b"
Nov 03 22:24:13.377678 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:13.377637    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3553842-86f5-4f08-bc2f-4e7c83e4ca0e-istio-podinfo\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b\" (UID: \"c3553842-86f5-4f08-bc2f-4e7c83e4ca0e\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b"
Nov 03 22:24:13.377801 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:13.377780    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3553842-86f5-4f08-bc2f-4e7c83e4ca0e-istio-envoy\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b\" (UID: \"c3553842-86f5-4f08-bc2f-4e7c83e4ca0e\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b"
Nov 03 22:24:13.377859 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:13.377815    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3553842-86f5-4f08-bc2f-4e7c83e4ca0e-istio-data\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b\" (UID: \"c3553842-86f5-4f08-bc2f-4e7c83e4ca0e\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b"
Nov 03 22:24:13.377859 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:13.377848    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3553842-86f5-4f08-bc2f-4e7c83e4ca0e-credential-socket\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b\" (UID: \"c3553842-86f5-4f08-bc2f-4e7c83e4ca0e\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b"
Nov 03 22:24:13.377934 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:13.377913    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3553842-86f5-4f08-bc2f-4e7c83e4ca0e-istio-token\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b\" (UID: \"c3553842-86f5-4f08-bc2f-4e7c83e4ca0e\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b"
Nov 03 22:24:13.377973 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:13.377949    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3553842-86f5-4f08-bc2f-4e7c83e4ca0e-workload-certs\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b\" (UID: \"c3553842-86f5-4f08-bc2f-4e7c83e4ca0e\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b"
Nov 03 22:24:13.378015 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:13.377972    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mw7f5\" (UniqueName: \"kubernetes.io/projected/c3553842-86f5-4f08-bc2f-4e7c83e4ca0e-kube-api-access-mw7f5\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b\" (UID: \"c3553842-86f5-4f08-bc2f-4e7c83e4ca0e\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b"
Nov 03 22:24:13.478988 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:13.478951    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3553842-86f5-4f08-bc2f-4e7c83e4ca0e-istio-token\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b\" (UID: \"c3553842-86f5-4f08-bc2f-4e7c83e4ca0e\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b"
Nov 03 22:24:13.479144 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:13.479000    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3553842-86f5-4f08-bc2f-4e7c83e4ca0e-workload-certs\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b\" (UID: \"c3553842-86f5-4f08-bc2f-4e7c83e4ca0e\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b"
Nov 03 22:24:13.479144 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:13.479027    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mw7f5\" (UniqueName: \"kubernetes.io/projected/c3553842-86f5-4f08-bc2f-4e7c83e4ca0e-kube-api-access-mw7f5\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b\" (UID: \"c3553842-86f5-4f08-bc2f-4e7c83e4ca0e\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b"
Nov 03 22:24:13.479144 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:13.479063    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3553842-86f5-4f08-bc2f-4e7c83e4ca0e-istiod-ca-cert\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b\" (UID: \"c3553842-86f5-4f08-bc2f-4e7c83e4ca0e\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b"
Nov 03 22:24:13.479144 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:13.479107    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3553842-86f5-4f08-bc2f-4e7c83e4ca0e-workload-socket\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b\" (UID: \"c3553842-86f5-4f08-bc2f-4e7c83e4ca0e\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b"
Nov 03 22:24:13.479144 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:13.479128    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3553842-86f5-4f08-bc2f-4e7c83e4ca0e-istio-podinfo\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b\" (UID: \"c3553842-86f5-4f08-bc2f-4e7c83e4ca0e\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b"
Nov 03 22:24:13.479488 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:13.479203    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3553842-86f5-4f08-bc2f-4e7c83e4ca0e-istio-envoy\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b\" (UID: \"c3553842-86f5-4f08-bc2f-4e7c83e4ca0e\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b"
Nov 03 22:24:13.479488 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:13.479259    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3553842-86f5-4f08-bc2f-4e7c83e4ca0e-istio-data\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b\" (UID: \"c3553842-86f5-4f08-bc2f-4e7c83e4ca0e\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b"
Nov 03 22:24:13.479488 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:13.479290    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3553842-86f5-4f08-bc2f-4e7c83e4ca0e-credential-socket\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b\" (UID: \"c3553842-86f5-4f08-bc2f-4e7c83e4ca0e\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b"
Nov 03 22:24:13.479660 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:13.479572    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3553842-86f5-4f08-bc2f-4e7c83e4ca0e-workload-certs\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b\" (UID: \"c3553842-86f5-4f08-bc2f-4e7c83e4ca0e\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b"
Nov 03 22:24:13.479825 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:13.479793    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3553842-86f5-4f08-bc2f-4e7c83e4ca0e-workload-socket\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b\" (UID: \"c3553842-86f5-4f08-bc2f-4e7c83e4ca0e\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b"
Nov 03 22:24:13.479931 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:13.479827    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3553842-86f5-4f08-bc2f-4e7c83e4ca0e-istio-data\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b\" (UID: \"c3553842-86f5-4f08-bc2f-4e7c83e4ca0e\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b"
Nov 03 22:24:13.479931 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:13.479797    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3553842-86f5-4f08-bc2f-4e7c83e4ca0e-credential-socket\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b\" (UID: \"c3553842-86f5-4f08-bc2f-4e7c83e4ca0e\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b"
Nov 03 22:24:13.480034 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:13.480013    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3553842-86f5-4f08-bc2f-4e7c83e4ca0e-istiod-ca-cert\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b\" (UID: \"c3553842-86f5-4f08-bc2f-4e7c83e4ca0e\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b"
Nov 03 22:24:13.482148 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:13.482118    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3553842-86f5-4f08-bc2f-4e7c83e4ca0e-istio-envoy\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b\" (UID: \"c3553842-86f5-4f08-bc2f-4e7c83e4ca0e\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b"
Nov 03 22:24:13.482266 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:13.482250    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3553842-86f5-4f08-bc2f-4e7c83e4ca0e-istio-podinfo\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b\" (UID: \"c3553842-86f5-4f08-bc2f-4e7c83e4ca0e\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b"
Nov 03 22:24:13.503642 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:13.503613    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:24:13.504976 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:13.504954    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3553842-86f5-4f08-bc2f-4e7c83e4ca0e-istio-token\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b\" (UID: \"c3553842-86f5-4f08-bc2f-4e7c83e4ca0e\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b"
Nov 03 22:24:13.505076 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:13.505061    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mw7f5\" (UniqueName: \"kubernetes.io/projected/c3553842-86f5-4f08-bc2f-4e7c83e4ca0e-kube-api-access-mw7f5\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b\" (UID: \"c3553842-86f5-4f08-bc2f-4e7c83e4ca0e\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b"
Nov 03 22:24:13.624165 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:13.624082    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b"
Nov 03 22:24:13.938444 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:13.938417    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b"]
Nov 03 22:24:13.940362 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:24:13.940333    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3553842_86f5_4f08_bc2f_4e7c83e4ca0e.slice/crio-a02b299bc47a72d3f801ec620a8ef9b98425160c522c3a983501c1c53313b960 WatchSource:0}: Error finding container a02b299bc47a72d3f801ec620a8ef9b98425160c522c3a983501c1c53313b960: Status 404 returned error can't find the container with id a02b299bc47a72d3f801ec620a8ef9b98425160c522c3a983501c1c53313b960
Nov 03 22:24:13.942613 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:13.942577    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:24:13.942718 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:13.942659    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:24:13.942718 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:13.942707    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:24:14.197532 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:14.197424    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b" event={"ID":"c3553842-86f5-4f08-bc2f-4e7c83e4ca0e","Type":"ContainerStarted","Data":"c19256fa8088619af1d8330696f0349c2c24ff1f8b28dd1ba694bc54c691fe1a"}
Nov 03 22:24:14.197532 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:14.197483    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b" event={"ID":"c3553842-86f5-4f08-bc2f-4e7c83e4ca0e","Type":"ContainerStarted","Data":"a02b299bc47a72d3f801ec620a8ef9b98425160c522c3a983501c1c53313b960"}
Nov 03 22:24:14.243609 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:14.243554    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b" podStartSLOduration=1.243538118 podStartE2EDuration="1.243538118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:24:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:24:14.241594966 +0000 UTC m=+678.267450840" watchObservedRunningTime="2026-11-03 22:24:14.243538118 +0000 UTC m=+678.269393980"
Nov 03 22:24:14.625204 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:14.625123    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b"
Nov 03 22:24:14.626770 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:14.626744    4431 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Nov 03 22:24:14.626891 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:14.626791    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b" podUID="c3553842-86f5-4f08-bc2f-4e7c83e4ca0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Nov 03 22:24:15.119202 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:15.119164    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:24:15.625193 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:15.625155    4431 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Nov 03 22:24:15.625394 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:15.625210    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b" podUID="c3553842-86f5-4f08-bc2f-4e7c83e4ca0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Nov 03 22:24:16.624569 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:16.624524    4431 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Nov 03 22:24:16.624779 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:16.624595    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b" podUID="c3553842-86f5-4f08-bc2f-4e7c83e4ca0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Nov 03 22:24:16.702685 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:16.702626    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:24:17.624614 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:17.624572    4431 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Nov 03 22:24:17.624809 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:17.624648    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b" podUID="c3553842-86f5-4f08-bc2f-4e7c83e4ca0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Nov 03 22:24:18.461988 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:18.461887    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:24:18.624535 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:18.624502    4431 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Nov 03 22:24:18.624754 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:18.624555    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b" podUID="c3553842-86f5-4f08-bc2f-4e7c83e4ca0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Nov 03 22:24:19.625284 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:19.625249    4431 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Nov 03 22:24:19.625680 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:19.625299    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b" podUID="c3553842-86f5-4f08-bc2f-4e7c83e4ca0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Nov 03 22:24:20.130997 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:20.130956    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:24:21.251897 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:21.251864    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b"
Nov 03 22:24:21.252374 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:21.252318    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b"
Nov 03 22:24:21.253609 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:21.253587    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b"
Nov 03 22:24:21.763856 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:21.763816    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:24:23.528989 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:23.528954    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:24:25.036419 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:25.036386    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:24:26.519154 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:26.519118    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:24:27.510100 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:27.510062    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp"]
Nov 03 22:24:27.515181 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:27.515160    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp"
Nov 03 22:24:27.532649 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:27.532626    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-50d738-openshift-default-dockercfg-jhlmb\""
Nov 03 22:24:27.544188 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:27.544160    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp"]
Nov 03 22:24:27.562860 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:27.562823    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1787f2c-7d0d-4bda-9a6d-f63a41b6b503-workload-certs\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp\" (UID: \"a1787f2c-7d0d-4bda-9a6d-f63a41b6b503\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp"
Nov 03 22:24:27.563034 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:27.562873    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1787f2c-7d0d-4bda-9a6d-f63a41b6b503-istio-podinfo\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp\" (UID: \"a1787f2c-7d0d-4bda-9a6d-f63a41b6b503\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp"
Nov 03 22:24:27.563034 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:27.562891    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwcck\" (UniqueName: \"kubernetes.io/projected/a1787f2c-7d0d-4bda-9a6d-f63a41b6b503-kube-api-access-xwcck\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp\" (UID: \"a1787f2c-7d0d-4bda-9a6d-f63a41b6b503\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp"
Nov 03 22:24:27.563034 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:27.562969    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1787f2c-7d0d-4bda-9a6d-f63a41b6b503-credential-socket\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp\" (UID: \"a1787f2c-7d0d-4bda-9a6d-f63a41b6b503\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp"
Nov 03 22:24:27.563234 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:27.563056    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1787f2c-7d0d-4bda-9a6d-f63a41b6b503-istio-envoy\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp\" (UID: \"a1787f2c-7d0d-4bda-9a6d-f63a41b6b503\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp"
Nov 03 22:24:27.563234 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:27.563164    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1787f2c-7d0d-4bda-9a6d-f63a41b6b503-istio-data\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp\" (UID: \"a1787f2c-7d0d-4bda-9a6d-f63a41b6b503\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp"
Nov 03 22:24:27.563234 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:27.563195    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1787f2c-7d0d-4bda-9a6d-f63a41b6b503-istiod-ca-cert\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp\" (UID: \"a1787f2c-7d0d-4bda-9a6d-f63a41b6b503\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp"
Nov 03 22:24:27.563408 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:27.563284    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1787f2c-7d0d-4bda-9a6d-f63a41b6b503-workload-socket\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp\" (UID: \"a1787f2c-7d0d-4bda-9a6d-f63a41b6b503\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp"
Nov 03 22:24:27.563408 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:27.563300    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1787f2c-7d0d-4bda-9a6d-f63a41b6b503-istio-token\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp\" (UID: \"a1787f2c-7d0d-4bda-9a6d-f63a41b6b503\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp"
Nov 03 22:24:27.664297 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:27.664251    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1787f2c-7d0d-4bda-9a6d-f63a41b6b503-workload-certs\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp\" (UID: \"a1787f2c-7d0d-4bda-9a6d-f63a41b6b503\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp"
Nov 03 22:24:27.664468 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:27.664309    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1787f2c-7d0d-4bda-9a6d-f63a41b6b503-istio-podinfo\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp\" (UID: \"a1787f2c-7d0d-4bda-9a6d-f63a41b6b503\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp"
Nov 03 22:24:27.664468 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:27.664330    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwcck\" (UniqueName: \"kubernetes.io/projected/a1787f2c-7d0d-4bda-9a6d-f63a41b6b503-kube-api-access-xwcck\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp\" (UID: \"a1787f2c-7d0d-4bda-9a6d-f63a41b6b503\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp"
Nov 03 22:24:27.664468 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:27.664356    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1787f2c-7d0d-4bda-9a6d-f63a41b6b503-credential-socket\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp\" (UID: \"a1787f2c-7d0d-4bda-9a6d-f63a41b6b503\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp"
Nov 03 22:24:27.664468 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:27.664391    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1787f2c-7d0d-4bda-9a6d-f63a41b6b503-istio-envoy\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp\" (UID: \"a1787f2c-7d0d-4bda-9a6d-f63a41b6b503\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp"
Nov 03 22:24:27.664643 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:27.664468    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1787f2c-7d0d-4bda-9a6d-f63a41b6b503-istio-data\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp\" (UID: \"a1787f2c-7d0d-4bda-9a6d-f63a41b6b503\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp"
Nov 03 22:24:27.664643 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:27.664512    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1787f2c-7d0d-4bda-9a6d-f63a41b6b503-istiod-ca-cert\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp\" (UID: \"a1787f2c-7d0d-4bda-9a6d-f63a41b6b503\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp"
Nov 03 22:24:27.664643 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:27.664575    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1787f2c-7d0d-4bda-9a6d-f63a41b6b503-workload-socket\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp\" (UID: \"a1787f2c-7d0d-4bda-9a6d-f63a41b6b503\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp"
Nov 03 22:24:27.664643 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:27.664600    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1787f2c-7d0d-4bda-9a6d-f63a41b6b503-istio-token\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp\" (UID: \"a1787f2c-7d0d-4bda-9a6d-f63a41b6b503\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp"
Nov 03 22:24:27.664804 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:27.664685    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1787f2c-7d0d-4bda-9a6d-f63a41b6b503-workload-certs\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp\" (UID: \"a1787f2c-7d0d-4bda-9a6d-f63a41b6b503\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp"
Nov 03 22:24:27.664804 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:27.664786    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1787f2c-7d0d-4bda-9a6d-f63a41b6b503-credential-socket\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp\" (UID: \"a1787f2c-7d0d-4bda-9a6d-f63a41b6b503\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp"
Nov 03 22:24:27.665107 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:27.665087    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1787f2c-7d0d-4bda-9a6d-f63a41b6b503-istiod-ca-cert\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp\" (UID: \"a1787f2c-7d0d-4bda-9a6d-f63a41b6b503\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp"
Nov 03 22:24:27.665366 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:27.665342    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1787f2c-7d0d-4bda-9a6d-f63a41b6b503-istio-data\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp\" (UID: \"a1787f2c-7d0d-4bda-9a6d-f63a41b6b503\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp"
Nov 03 22:24:27.665504 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:27.665487    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1787f2c-7d0d-4bda-9a6d-f63a41b6b503-workload-socket\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp\" (UID: \"a1787f2c-7d0d-4bda-9a6d-f63a41b6b503\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp"
Nov 03 22:24:27.666778 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:27.666759    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1787f2c-7d0d-4bda-9a6d-f63a41b6b503-istio-podinfo\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp\" (UID: \"a1787f2c-7d0d-4bda-9a6d-f63a41b6b503\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp"
Nov 03 22:24:27.667096 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:27.667073    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1787f2c-7d0d-4bda-9a6d-f63a41b6b503-istio-envoy\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp\" (UID: \"a1787f2c-7d0d-4bda-9a6d-f63a41b6b503\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp"
Nov 03 22:24:27.691926 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:27.691893    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9k6cb"]
Nov 03 22:24:27.696081 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:27.696059    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9k6cb"
Nov 03 22:24:27.696873 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:27.696846    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwcck\" (UniqueName: \"kubernetes.io/projected/a1787f2c-7d0d-4bda-9a6d-f63a41b6b503-kube-api-access-xwcck\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp\" (UID: \"a1787f2c-7d0d-4bda-9a6d-f63a41b6b503\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp"
Nov 03 22:24:27.701736 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:27.701708    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1787f2c-7d0d-4bda-9a6d-f63a41b6b503-istio-token\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp\" (UID: \"a1787f2c-7d0d-4bda-9a6d-f63a41b6b503\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp"
Nov 03 22:24:27.720686 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:27.720658    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9k6cb"]
Nov 03 22:24:27.765920 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:27.765840    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e568c172-7301-4d7f-9499-6c3a1619c9dd-istio-data\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-9k6cb\" (UID: \"e568c172-7301-4d7f-9499-6c3a1619c9dd\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9k6cb"
Nov 03 22:24:27.765920 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:27.765895    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e568c172-7301-4d7f-9499-6c3a1619c9dd-credential-socket\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-9k6cb\" (UID: \"e568c172-7301-4d7f-9499-6c3a1619c9dd\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9k6cb"
Nov 03 22:24:27.766109 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:27.765982    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwsqn\" (UniqueName: \"kubernetes.io/projected/e568c172-7301-4d7f-9499-6c3a1619c9dd-kube-api-access-kwsqn\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-9k6cb\" (UID: \"e568c172-7301-4d7f-9499-6c3a1619c9dd\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9k6cb"
Nov 03 22:24:27.766109 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:27.766013    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e568c172-7301-4d7f-9499-6c3a1619c9dd-istio-envoy\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-9k6cb\" (UID: \"e568c172-7301-4d7f-9499-6c3a1619c9dd\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9k6cb"
Nov 03 22:24:27.766109 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:27.766033    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e568c172-7301-4d7f-9499-6c3a1619c9dd-workload-certs\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-9k6cb\" (UID: \"e568c172-7301-4d7f-9499-6c3a1619c9dd\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9k6cb"
Nov 03 22:24:27.766243 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:27.766122    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e568c172-7301-4d7f-9499-6c3a1619c9dd-istiod-ca-cert\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-9k6cb\" (UID: \"e568c172-7301-4d7f-9499-6c3a1619c9dd\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9k6cb"
Nov 03 22:24:27.766243 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:27.766193    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e568c172-7301-4d7f-9499-6c3a1619c9dd-istio-token\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-9k6cb\" (UID: \"e568c172-7301-4d7f-9499-6c3a1619c9dd\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9k6cb"
Nov 03 22:24:27.766319 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:27.766242    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e568c172-7301-4d7f-9499-6c3a1619c9dd-istio-podinfo\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-9k6cb\" (UID: \"e568c172-7301-4d7f-9499-6c3a1619c9dd\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9k6cb"
Nov 03 22:24:27.766319 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:27.766289    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e568c172-7301-4d7f-9499-6c3a1619c9dd-workload-socket\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-9k6cb\" (UID: \"e568c172-7301-4d7f-9499-6c3a1619c9dd\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9k6cb"
Nov 03 22:24:27.796691 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:27.796661    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-553753-openshift-default-dockercfg-vgskd\""
Nov 03 22:24:27.829784 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:27.829756    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp"
Nov 03 22:24:27.867830 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:27.867799    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e568c172-7301-4d7f-9499-6c3a1619c9dd-credential-socket\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-9k6cb\" (UID: \"e568c172-7301-4d7f-9499-6c3a1619c9dd\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9k6cb"
Nov 03 22:24:27.868006 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:27.867848    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwsqn\" (UniqueName: \"kubernetes.io/projected/e568c172-7301-4d7f-9499-6c3a1619c9dd-kube-api-access-kwsqn\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-9k6cb\" (UID: \"e568c172-7301-4d7f-9499-6c3a1619c9dd\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9k6cb"
Nov 03 22:24:27.868006 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:27.867949    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e568c172-7301-4d7f-9499-6c3a1619c9dd-istio-envoy\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-9k6cb\" (UID: \"e568c172-7301-4d7f-9499-6c3a1619c9dd\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9k6cb"
Nov 03 22:24:27.868006 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:27.867975    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e568c172-7301-4d7f-9499-6c3a1619c9dd-workload-certs\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-9k6cb\" (UID: \"e568c172-7301-4d7f-9499-6c3a1619c9dd\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9k6cb"
Nov 03 22:24:27.868006 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:27.868002    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e568c172-7301-4d7f-9499-6c3a1619c9dd-istiod-ca-cert\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-9k6cb\" (UID: \"e568c172-7301-4d7f-9499-6c3a1619c9dd\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9k6cb"
Nov 03 22:24:27.868259 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:27.868055    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e568c172-7301-4d7f-9499-6c3a1619c9dd-istio-token\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-9k6cb\" (UID: \"e568c172-7301-4d7f-9499-6c3a1619c9dd\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9k6cb"
Nov 03 22:24:27.868259 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:27.868099    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e568c172-7301-4d7f-9499-6c3a1619c9dd-istio-podinfo\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-9k6cb\" (UID: \"e568c172-7301-4d7f-9499-6c3a1619c9dd\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9k6cb"
Nov 03 22:24:27.868259 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:27.868156    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e568c172-7301-4d7f-9499-6c3a1619c9dd-workload-socket\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-9k6cb\" (UID: \"e568c172-7301-4d7f-9499-6c3a1619c9dd\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9k6cb"
Nov 03 22:24:27.868259 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:27.868228    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e568c172-7301-4d7f-9499-6c3a1619c9dd-istio-data\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-9k6cb\" (UID: \"e568c172-7301-4d7f-9499-6c3a1619c9dd\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9k6cb"
Nov 03 22:24:27.868475 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:27.868340    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e568c172-7301-4d7f-9499-6c3a1619c9dd-credential-socket\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-9k6cb\" (UID: \"e568c172-7301-4d7f-9499-6c3a1619c9dd\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9k6cb"
Nov 03 22:24:27.868475 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:27.868413    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e568c172-7301-4d7f-9499-6c3a1619c9dd-workload-certs\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-9k6cb\" (UID: \"e568c172-7301-4d7f-9499-6c3a1619c9dd\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9k6cb"
Nov 03 22:24:27.868647 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:27.868623    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e568c172-7301-4d7f-9499-6c3a1619c9dd-istio-data\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-9k6cb\" (UID: \"e568c172-7301-4d7f-9499-6c3a1619c9dd\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9k6cb"
Nov 03 22:24:27.868753 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:27.868732    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e568c172-7301-4d7f-9499-6c3a1619c9dd-istiod-ca-cert\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-9k6cb\" (UID: \"e568c172-7301-4d7f-9499-6c3a1619c9dd\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9k6cb"
Nov 03 22:24:27.868896 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:27.868874    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e568c172-7301-4d7f-9499-6c3a1619c9dd-workload-socket\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-9k6cb\" (UID: \"e568c172-7301-4d7f-9499-6c3a1619c9dd\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9k6cb"
Nov 03 22:24:27.870819 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:27.870763    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e568c172-7301-4d7f-9499-6c3a1619c9dd-istio-envoy\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-9k6cb\" (UID: \"e568c172-7301-4d7f-9499-6c3a1619c9dd\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9k6cb"
Nov 03 22:24:27.871098 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:27.871081    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e568c172-7301-4d7f-9499-6c3a1619c9dd-istio-podinfo\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-9k6cb\" (UID: \"e568c172-7301-4d7f-9499-6c3a1619c9dd\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9k6cb"
Nov 03 22:24:27.892224 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:27.892177    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwsqn\" (UniqueName: \"kubernetes.io/projected/e568c172-7301-4d7f-9499-6c3a1619c9dd-kube-api-access-kwsqn\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-9k6cb\" (UID: \"e568c172-7301-4d7f-9499-6c3a1619c9dd\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9k6cb"
Nov 03 22:24:27.894753 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:27.894722    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e568c172-7301-4d7f-9499-6c3a1619c9dd-istio-token\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-9k6cb\" (UID: \"e568c172-7301-4d7f-9499-6c3a1619c9dd\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9k6cb"
Nov 03 22:24:28.033794 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:28.033714    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9k6cb"
Nov 03 22:24:28.145472 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:28.145443    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp"]
Nov 03 22:24:28.148372 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:24:28.148343    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1787f2c_7d0d_4bda_9a6d_f63a41b6b503.slice/crio-ec14cf58b0de06ddd7beff974f36f1acac21cd7b346fa5d5493155adebeae496 WatchSource:0}: Error finding container ec14cf58b0de06ddd7beff974f36f1acac21cd7b346fa5d5493155adebeae496: Status 404 returned error can't find the container with id ec14cf58b0de06ddd7beff974f36f1acac21cd7b346fa5d5493155adebeae496
Nov 03 22:24:28.150754 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:28.150722    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:24:28.150829 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:28.150789    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:24:28.150829 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:28.150818    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:24:28.216740 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:28.216711    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:24:28.271967 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:28.271926    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp" event={"ID":"a1787f2c-7d0d-4bda-9a6d-f63a41b6b503","Type":"ContainerStarted","Data":"dc68de302c9f48dea33d7402dfac812486e5bb73bca24fa5852ec8b7bb84b123"}
Nov 03 22:24:28.272152 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:28.271993    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp" event={"ID":"a1787f2c-7d0d-4bda-9a6d-f63a41b6b503","Type":"ContainerStarted","Data":"ec14cf58b0de06ddd7beff974f36f1acac21cd7b346fa5d5493155adebeae496"}
Nov 03 22:24:28.320523 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:28.320421    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp" podStartSLOduration=1.320407552 podStartE2EDuration="1.320407552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:24:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:24:28.317853271 +0000 UTC m=+692.343709133" watchObservedRunningTime="2026-11-03 22:24:28.320407552 +0000 UTC m=+692.346263414"
Nov 03 22:24:28.336699 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:28.336669    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9k6cb"]
Nov 03 22:24:28.338460 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:24:28.338434    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode568c172_7301_4d7f_9499_6c3a1619c9dd.slice/crio-7be113d7f4e432c6a17d4b18deb2b5268782e62cec40f7ab0d8598f76d89eb33 WatchSource:0}: Error finding container 7be113d7f4e432c6a17d4b18deb2b5268782e62cec40f7ab0d8598f76d89eb33: Status 404 returned error can't find the container with id 7be113d7f4e432c6a17d4b18deb2b5268782e62cec40f7ab0d8598f76d89eb33
Nov 03 22:24:28.341557 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:28.341525    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:24:28.341637 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:28.341594    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:24:28.341637 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:28.341622    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:24:28.830734 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:28.830689    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp"
Nov 03 22:24:29.291506 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:29.291463    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9k6cb" event={"ID":"e568c172-7301-4d7f-9499-6c3a1619c9dd","Type":"ContainerStarted","Data":"9dcc65475817e4e163a86b4a88c69cd79305fa4f7c6b6739cb614a0aa7b4e416"}
Nov 03 22:24:29.291723 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:29.291536    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9k6cb" event={"ID":"e568c172-7301-4d7f-9499-6c3a1619c9dd","Type":"ContainerStarted","Data":"7be113d7f4e432c6a17d4b18deb2b5268782e62cec40f7ab0d8598f76d89eb33"}
Nov 03 22:24:29.365240 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:29.361908    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9k6cb" podStartSLOduration=2.361887285 podStartE2EDuration="2.361887285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:24:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:24:29.357889247 +0000 UTC m=+693.383745109" watchObservedRunningTime="2026-11-03 22:24:29.361887285 +0000 UTC m=+693.387743153"
Nov 03 22:24:29.616290 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:29.616191    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:24:29.832510 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:29.832460    4431 patch_prober.go:29] interesting pod/e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Nov 03 22:24:29.832867 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:29.832536    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp" podUID="a1787f2c-7d0d-4bda-9a6d-f63a41b6b503" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Nov 03 22:24:29.837113 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:29.837089    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:24:30.034572 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:30.034541    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9k6cb"
Nov 03 22:24:30.036389 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:30.036251    4431 patch_prober.go:29] interesting pod/e2e-worker-w2-553753-openshift-default-558949869c-9k6cb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Nov 03 22:24:30.036389 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:30.036300    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9k6cb" podUID="e568c172-7301-4d7f-9499-6c3a1619c9dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Nov 03 22:24:30.830449 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:30.830414    4431 patch_prober.go:29] interesting pod/e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Nov 03 22:24:30.830626 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:30.830478    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp" podUID="a1787f2c-7d0d-4bda-9a6d-f63a41b6b503" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Nov 03 22:24:31.034511 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:31.034467    4431 patch_prober.go:29] interesting pod/e2e-worker-w2-553753-openshift-default-558949869c-9k6cb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Nov 03 22:24:31.034879 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:31.034530    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9k6cb" podUID="e568c172-7301-4d7f-9499-6c3a1619c9dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Nov 03 22:24:31.304993 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:24:31.304959    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:24:31.565208 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:31.565106    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:24:31.830824 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:31.830737    4431 patch_prober.go:29] interesting pod/e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Nov 03 22:24:31.830824 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:31.830804    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp" podUID="a1787f2c-7d0d-4bda-9a6d-f63a41b6b503" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Nov 03 22:24:32.034125 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:32.034086    4431 patch_prober.go:29] interesting pod/e2e-worker-w2-553753-openshift-default-558949869c-9k6cb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Nov 03 22:24:32.034321 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:32.034139    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9k6cb" podUID="e568c172-7301-4d7f-9499-6c3a1619c9dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Nov 03 22:24:32.830515 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:32.830484    4431 patch_prober.go:29] interesting pod/e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Nov 03 22:24:32.830902 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:32.830538    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp" podUID="a1787f2c-7d0d-4bda-9a6d-f63a41b6b503" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Nov 03 22:24:33.034595 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:33.034556    4431 patch_prober.go:29] interesting pod/e2e-worker-w2-553753-openshift-default-558949869c-9k6cb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Nov 03 22:24:33.034795 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:33.034616    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9k6cb" podUID="e568c172-7301-4d7f-9499-6c3a1619c9dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Nov 03 22:24:33.459662 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:33.459625    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:24:33.830793 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:33.830703    4431 patch_prober.go:29] interesting pod/e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Nov 03 22:24:33.830793 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:33.830762    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp" podUID="a1787f2c-7d0d-4bda-9a6d-f63a41b6b503" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Nov 03 22:24:34.034689 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:34.034641    4431 patch_prober.go:29] interesting pod/e2e-worker-w2-553753-openshift-default-558949869c-9k6cb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Nov 03 22:24:34.034889 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:34.034708    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9k6cb" podUID="e568c172-7301-4d7f-9499-6c3a1619c9dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Nov 03 22:24:34.904530 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:34.904492    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp"
Nov 03 22:24:34.905860 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:34.905837    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp"
Nov 03 22:24:34.906733 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:34.906689    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp"
Nov 03 22:24:35.080285 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:35.080255    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9k6cb"
Nov 03 22:24:35.081395 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:35.081279    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9k6cb"
Nov 03 22:24:35.083187 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:35.083166    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9k6cb"
Nov 03 22:24:35.137992 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:35.137957    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:24:36.622364 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:36.622334    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:24:38.109101 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:38.109064    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:24:39.714073 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:39.714039    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:24:41.308075 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:41.308033    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:24:42.909647 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:42.909613    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:24:44.610776 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:44.610728    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:24:45.245537 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:45.245497    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:24:46.327647 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:46.327616    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:24:47.896509 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:47.896472    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:24:49.418825 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:49.418790    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:24:50.916713 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:50.916671    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:24:52.405562 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:52.405526    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:24:53.910056 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:53.910021    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:24:55.521632 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:55.521601    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:24:57.092410 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:57.092375    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:24:58.591573 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:24:58.591537    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:25:00.094902 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:00.094867    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:25:01.258832 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:25:01.258789    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:25:01.673473 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:01.673440    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:25:03.158167 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:03.158129    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:25:04.630056 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:04.630024    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:25:06.294868 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:06.294835    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:25:07.932970 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:07.932934    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:25:09.628307 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:09.628266    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:25:11.382343 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:11.382269    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:25:13.116245 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:13.116193    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:25:14.703896 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:14.703830    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:25:16.408536 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:16.408496    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:25:18.005025 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:18.004990    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:25:19.604885 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:19.604851    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:25:21.119161 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:21.119118    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:25:22.808924 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:22.808886    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:25:23.204444 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:23.204390    4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-r4qjb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 22:25:23.204661 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:23.204461    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-r4qjb" podUID="d5687c54-7c44-4806-846e-3646f096b7ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:25:24.668952 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:24.668916    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:25:26.273727 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:26.273689    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:25:27.924555 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:27.924517    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:25:29.517417 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:29.517383    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:25:31.222022 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:31.221980    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:25:31.261735 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:25:31.261695    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:25:31.374552 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:31.374501    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:25:33.092293 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:33.092196    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:25:34.731163 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:34.731126    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:25:36.554027 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:36.553991    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:25:38.213403 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:38.213339    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:25:39.817878 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:39.817844    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:25:41.317819 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:41.317786    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:25:42.801204 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:42.801159    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:25:44.135082 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:44.135044    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-crcsv"]
Nov 03 22:25:44.139022 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:44.138991    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-crcsv"
Nov 03 22:25:44.172240 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:44.172181    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-crcsv"]
Nov 03 22:25:44.250575 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:44.250543    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f5907c4-80f0-4055-b561-30650efcbb92-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-crcsv\" (UID: \"3f5907c4-80f0-4055-b561-30650efcbb92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-crcsv"
Nov 03 22:25:44.250760 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:44.250601    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f5907c4-80f0-4055-b561-30650efcbb92-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-crcsv\" (UID: \"3f5907c4-80f0-4055-b561-30650efcbb92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-crcsv"
Nov 03 22:25:44.250760 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:44.250674    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f5907c4-80f0-4055-b561-30650efcbb92-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-crcsv\" (UID: \"3f5907c4-80f0-4055-b561-30650efcbb92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-crcsv"
Nov 03 22:25:44.250760 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:44.250721    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f5907c4-80f0-4055-b561-30650efcbb92-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-crcsv\" (UID: \"3f5907c4-80f0-4055-b561-30650efcbb92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-crcsv"
Nov 03 22:25:44.250760 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:44.250743    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f5907c4-80f0-4055-b561-30650efcbb92-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-crcsv\" (UID: \"3f5907c4-80f0-4055-b561-30650efcbb92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-crcsv"
Nov 03 22:25:44.250985 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:44.250872    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f5907c4-80f0-4055-b561-30650efcbb92-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-crcsv\" (UID: \"3f5907c4-80f0-4055-b561-30650efcbb92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-crcsv"
Nov 03 22:25:44.250985 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:44.250902    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f5907c4-80f0-4055-b561-30650efcbb92-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-crcsv\" (UID: \"3f5907c4-80f0-4055-b561-30650efcbb92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-crcsv"
Nov 03 22:25:44.250985 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:44.250936    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f5907c4-80f0-4055-b561-30650efcbb92-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-crcsv\" (UID: \"3f5907c4-80f0-4055-b561-30650efcbb92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-crcsv"
Nov 03 22:25:44.251137 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:44.251032    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pktfd\" (UniqueName: \"kubernetes.io/projected/3f5907c4-80f0-4055-b561-30650efcbb92-kube-api-access-pktfd\") pod \"maas-default-gateway-openshift-default-7df854bf67-crcsv\" (UID: \"3f5907c4-80f0-4055-b561-30650efcbb92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-crcsv"
Nov 03 22:25:44.352007 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:44.351968    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f5907c4-80f0-4055-b561-30650efcbb92-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-crcsv\" (UID: \"3f5907c4-80f0-4055-b561-30650efcbb92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-crcsv"
Nov 03 22:25:44.352196 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:44.352020    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f5907c4-80f0-4055-b561-30650efcbb92-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-crcsv\" (UID: \"3f5907c4-80f0-4055-b561-30650efcbb92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-crcsv"
Nov 03 22:25:44.352196 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:44.352036    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f5907c4-80f0-4055-b561-30650efcbb92-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-crcsv\" (UID: \"3f5907c4-80f0-4055-b561-30650efcbb92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-crcsv"
Nov 03 22:25:44.352196 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:44.352076    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f5907c4-80f0-4055-b561-30650efcbb92-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-crcsv\" (UID: \"3f5907c4-80f0-4055-b561-30650efcbb92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-crcsv"
Nov 03 22:25:44.352196 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:44.352095    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f5907c4-80f0-4055-b561-30650efcbb92-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-crcsv\" (UID: \"3f5907c4-80f0-4055-b561-30650efcbb92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-crcsv"
Nov 03 22:25:44.352196 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:44.352124    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f5907c4-80f0-4055-b561-30650efcbb92-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-crcsv\" (UID: \"3f5907c4-80f0-4055-b561-30650efcbb92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-crcsv"
Nov 03 22:25:44.352196 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:44.352140    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f5907c4-80f0-4055-b561-30650efcbb92-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-crcsv\" (UID: \"3f5907c4-80f0-4055-b561-30650efcbb92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-crcsv"
Nov 03 22:25:44.352553 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:44.352263    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f5907c4-80f0-4055-b561-30650efcbb92-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-crcsv\" (UID: \"3f5907c4-80f0-4055-b561-30650efcbb92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-crcsv"
Nov 03 22:25:44.352553 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:44.352323    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pktfd\" (UniqueName: \"kubernetes.io/projected/3f5907c4-80f0-4055-b561-30650efcbb92-kube-api-access-pktfd\") pod \"maas-default-gateway-openshift-default-7df854bf67-crcsv\" (UID: \"3f5907c4-80f0-4055-b561-30650efcbb92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-crcsv"
Nov 03 22:25:44.352553 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:44.352541    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f5907c4-80f0-4055-b561-30650efcbb92-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-crcsv\" (UID: \"3f5907c4-80f0-4055-b561-30650efcbb92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-crcsv"
Nov 03 22:25:44.352721 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:44.352640    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f5907c4-80f0-4055-b561-30650efcbb92-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-crcsv\" (UID: \"3f5907c4-80f0-4055-b561-30650efcbb92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-crcsv"
Nov 03 22:25:44.352721 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:44.352712    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f5907c4-80f0-4055-b561-30650efcbb92-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-crcsv\" (UID: \"3f5907c4-80f0-4055-b561-30650efcbb92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-crcsv"
Nov 03 22:25:44.352840 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:44.352814    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f5907c4-80f0-4055-b561-30650efcbb92-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-crcsv\" (UID: \"3f5907c4-80f0-4055-b561-30650efcbb92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-crcsv"
Nov 03 22:25:44.353103 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:44.352944    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f5907c4-80f0-4055-b561-30650efcbb92-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-crcsv\" (UID: \"3f5907c4-80f0-4055-b561-30650efcbb92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-crcsv"
Nov 03 22:25:44.354762 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:44.354737    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f5907c4-80f0-4055-b561-30650efcbb92-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-crcsv\" (UID: \"3f5907c4-80f0-4055-b561-30650efcbb92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-crcsv"
Nov 03 22:25:44.355293 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:44.355277    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f5907c4-80f0-4055-b561-30650efcbb92-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-crcsv\" (UID: \"3f5907c4-80f0-4055-b561-30650efcbb92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-crcsv"
Nov 03 22:25:44.375692 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:44.375666    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f5907c4-80f0-4055-b561-30650efcbb92-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-crcsv\" (UID: \"3f5907c4-80f0-4055-b561-30650efcbb92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-crcsv"
Nov 03 22:25:44.375823 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:44.375774    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pktfd\" (UniqueName: \"kubernetes.io/projected/3f5907c4-80f0-4055-b561-30650efcbb92-kube-api-access-pktfd\") pod \"maas-default-gateway-openshift-default-7df854bf67-crcsv\" (UID: \"3f5907c4-80f0-4055-b561-30650efcbb92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-crcsv"
Nov 03 22:25:44.446760 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:44.446729    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:25:44.452791 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:44.452765    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-crcsv"
Nov 03 22:25:44.771704 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:44.771673    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-crcsv"]
Nov 03 22:25:44.773310 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:25:44.773278    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f5907c4_80f0_4055_b561_30650efcbb92.slice/crio-c2a70ba29b46f05fc27459b96a3d8d740f47fea5c5d2d94f28d643b43668cd3a WatchSource:0}: Error finding container c2a70ba29b46f05fc27459b96a3d8d740f47fea5c5d2d94f28d643b43668cd3a: Status 404 returned error can't find the container with id c2a70ba29b46f05fc27459b96a3d8d740f47fea5c5d2d94f28d643b43668cd3a
Nov 03 22:25:44.775583 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:44.775552    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:25:44.775643 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:44.775616    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:25:44.775687 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:44.775644    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:25:45.454890 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:45.454852    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:25:45.708092 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:45.708006    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-crcsv" event={"ID":"3f5907c4-80f0-4055-b561-30650efcbb92","Type":"ContainerStarted","Data":"e37736308e2700ce41775f54017c9d0088d0960d98acc5d0bcf3c586deafed68"}
Nov 03 22:25:45.708348 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:45.708327    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-crcsv" event={"ID":"3f5907c4-80f0-4055-b561-30650efcbb92","Type":"ContainerStarted","Data":"c2a70ba29b46f05fc27459b96a3d8d740f47fea5c5d2d94f28d643b43668cd3a"}
Nov 03 22:25:46.336464 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:46.336413    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:25:46.460315 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:46.460282    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-crcsv"
Nov 03 22:25:46.464372 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:46.464326    4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-crcsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Nov 03 22:25:46.464610 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:46.464575    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-crcsv" podUID="3f5907c4-80f0-4055-b561-30650efcbb92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Nov 03 22:25:47.457860 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:47.457814    4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-crcsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Nov 03 22:25:47.458184 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:47.457885    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-crcsv" podUID="3f5907c4-80f0-4055-b561-30650efcbb92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Nov 03 22:25:48.118076 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:48.118040    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:25:48.454642 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:48.454606    4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-crcsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Nov 03 22:25:48.454946 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:48.454918    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-crcsv" podUID="3f5907c4-80f0-4055-b561-30650efcbb92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Nov 03 22:25:49.455240 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:49.453638    4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-crcsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Nov 03 22:25:49.455240 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:49.453715    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-crcsv" podUID="3f5907c4-80f0-4055-b561-30650efcbb92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Nov 03 22:25:49.821601 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:49.821503    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:25:50.247240 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:50.246723    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:25:50.457241 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:50.454588    4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-crcsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Nov 03 22:25:50.457241 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:50.454664    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-crcsv" podUID="3f5907c4-80f0-4055-b561-30650efcbb92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Nov 03 22:25:51.417432 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:51.417391    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:25:51.453295 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:51.453265    4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-crcsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Nov 03 22:25:51.453468 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:51.453318    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-crcsv" podUID="3f5907c4-80f0-4055-b561-30650efcbb92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Nov 03 22:25:52.905348 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:52.905283    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:25:53.242572 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:53.242528    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-crcsv"
Nov 03 22:25:53.242988 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:53.242963    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-crcsv"
Nov 03 22:25:53.243804 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:53.243784    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-crcsv"
Nov 03 22:25:53.301082 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:53.301019    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-crcsv" podStartSLOduration=9.301005097 podStartE2EDuration="9.301005097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:25:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:25:45.769391855 +0000 UTC m=+769.795247724" watchObservedRunningTime="2026-11-03 22:25:53.301005097 +0000 UTC m=+777.326860959"
Nov 03 22:25:53.508612 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:53.508526    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:25:54.518995 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:54.518966    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:25:56.112396 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:56.112362    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:25:57.777195 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:57.777162    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:25:58.284941 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:58.284903    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh"]
Nov 03 22:25:58.289072 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:58.289046    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh"
Nov 03 22:25:58.303634 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:58.303604    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-5187e1-openshift-default-dockercfg-6g9br\""
Nov 03 22:25:58.319609 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:58.319577    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh"]
Nov 03 22:25:58.361543 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:58.361512    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6855011d-1f9e-43be-8557-991f928f379e-workload-socket\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh\" (UID: \"6855011d-1f9e-43be-8557-991f928f379e\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh"
Nov 03 22:25:58.361752 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:58.361560    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5c5r5\" (UniqueName: \"kubernetes.io/projected/6855011d-1f9e-43be-8557-991f928f379e-kube-api-access-5c5r5\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh\" (UID: \"6855011d-1f9e-43be-8557-991f928f379e\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh"
Nov 03 22:25:58.361752 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:58.361703    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6855011d-1f9e-43be-8557-991f928f379e-workload-certs\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh\" (UID: \"6855011d-1f9e-43be-8557-991f928f379e\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh"
Nov 03 22:25:58.361752 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:58.361730    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6855011d-1f9e-43be-8557-991f928f379e-istio-data\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh\" (UID: \"6855011d-1f9e-43be-8557-991f928f379e\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh"
Nov 03 22:25:58.361752 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:58.361752    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6855011d-1f9e-43be-8557-991f928f379e-istio-envoy\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh\" (UID: \"6855011d-1f9e-43be-8557-991f928f379e\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh"
Nov 03 22:25:58.361998 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:58.361812    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6855011d-1f9e-43be-8557-991f928f379e-istiod-ca-cert\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh\" (UID: \"6855011d-1f9e-43be-8557-991f928f379e\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh"
Nov 03 22:25:58.361998 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:58.361857    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6855011d-1f9e-43be-8557-991f928f379e-credential-socket\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh\" (UID: \"6855011d-1f9e-43be-8557-991f928f379e\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh"
Nov 03 22:25:58.361998 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:58.361883    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6855011d-1f9e-43be-8557-991f928f379e-istio-podinfo\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh\" (UID: \"6855011d-1f9e-43be-8557-991f928f379e\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh"
Nov 03 22:25:58.361998 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:58.361961    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6855011d-1f9e-43be-8557-991f928f379e-istio-token\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh\" (UID: \"6855011d-1f9e-43be-8557-991f928f379e\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh"
Nov 03 22:25:58.463634 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:58.463594    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6855011d-1f9e-43be-8557-991f928f379e-workload-certs\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh\" (UID: \"6855011d-1f9e-43be-8557-991f928f379e\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh"
Nov 03 22:25:58.464356 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:58.464330    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6855011d-1f9e-43be-8557-991f928f379e-istio-data\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh\" (UID: \"6855011d-1f9e-43be-8557-991f928f379e\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh"
Nov 03 22:25:58.464555 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:58.464538    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6855011d-1f9e-43be-8557-991f928f379e-istio-envoy\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh\" (UID: \"6855011d-1f9e-43be-8557-991f928f379e\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh"
Nov 03 22:25:58.464717 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:58.464689    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6855011d-1f9e-43be-8557-991f928f379e-istiod-ca-cert\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh\" (UID: \"6855011d-1f9e-43be-8557-991f928f379e\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh"
Nov 03 22:25:58.464849 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:58.464834    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6855011d-1f9e-43be-8557-991f928f379e-credential-socket\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh\" (UID: \"6855011d-1f9e-43be-8557-991f928f379e\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh"
Nov 03 22:25:58.464972 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:58.464955    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6855011d-1f9e-43be-8557-991f928f379e-istio-podinfo\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh\" (UID: \"6855011d-1f9e-43be-8557-991f928f379e\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh"
Nov 03 22:25:58.465115 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:58.465101    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6855011d-1f9e-43be-8557-991f928f379e-istio-token\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh\" (UID: \"6855011d-1f9e-43be-8557-991f928f379e\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh"
Nov 03 22:25:58.465352 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:58.465306    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6855011d-1f9e-43be-8557-991f928f379e-workload-socket\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh\" (UID: \"6855011d-1f9e-43be-8557-991f928f379e\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh"
Nov 03 22:25:58.467758 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:58.467713    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c5r5\" (UniqueName: \"kubernetes.io/projected/6855011d-1f9e-43be-8557-991f928f379e-kube-api-access-5c5r5\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh\" (UID: \"6855011d-1f9e-43be-8557-991f928f379e\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh"
Nov 03 22:25:58.469798 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:58.464270    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6855011d-1f9e-43be-8557-991f928f379e-workload-certs\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh\" (UID: \"6855011d-1f9e-43be-8557-991f928f379e\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh"
Nov 03 22:25:58.469952 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:58.465970    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6855011d-1f9e-43be-8557-991f928f379e-workload-socket\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh\" (UID: \"6855011d-1f9e-43be-8557-991f928f379e\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh"
Nov 03 22:25:58.470077 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:58.467551    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6855011d-1f9e-43be-8557-991f928f379e-credential-socket\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh\" (UID: \"6855011d-1f9e-43be-8557-991f928f379e\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh"
Nov 03 22:25:58.470186 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:58.464697    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6855011d-1f9e-43be-8557-991f928f379e-istio-data\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh\" (UID: \"6855011d-1f9e-43be-8557-991f928f379e\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh"
Nov 03 22:25:58.470343 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:58.465720    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6855011d-1f9e-43be-8557-991f928f379e-istiod-ca-cert\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh\" (UID: \"6855011d-1f9e-43be-8557-991f928f379e\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh"
Nov 03 22:25:58.470343 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:58.469079    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6855011d-1f9e-43be-8557-991f928f379e-istio-envoy\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh\" (UID: \"6855011d-1f9e-43be-8557-991f928f379e\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh"
Nov 03 22:25:58.477296 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:58.473046    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6855011d-1f9e-43be-8557-991f928f379e-istio-podinfo\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh\" (UID: \"6855011d-1f9e-43be-8557-991f928f379e\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh"
Nov 03 22:25:58.495118 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:58.495053    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6855011d-1f9e-43be-8557-991f928f379e-istio-token\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh\" (UID: \"6855011d-1f9e-43be-8557-991f928f379e\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh"
Nov 03 22:25:58.499878 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:58.499819    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5c5r5\" (UniqueName: \"kubernetes.io/projected/6855011d-1f9e-43be-8557-991f928f379e-kube-api-access-5c5r5\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh\" (UID: \"6855011d-1f9e-43be-8557-991f928f379e\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh"
Nov 03 22:25:58.604586 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:58.604501    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh"
Nov 03 22:25:58.940609 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:58.940573    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh"]
Nov 03 22:25:58.946066 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:25:58.946033    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6855011d_1f9e_43be_8557_991f928f379e.slice/crio-7f1553beb8de03809ddb93d30a7990bdec51366a28caae39f791b8fa48e38fd6 WatchSource:0}: Error finding container 7f1553beb8de03809ddb93d30a7990bdec51366a28caae39f791b8fa48e38fd6: Status 404 returned error can't find the container with id 7f1553beb8de03809ddb93d30a7990bdec51366a28caae39f791b8fa48e38fd6
Nov 03 22:25:58.948876 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:58.948841    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:25:58.949004 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:58.948909    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:25:58.949004 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:58.948938    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:25:59.514995 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:59.514961    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:25:59.793131 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:59.793040    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh" event={"ID":"6855011d-1f9e-43be-8557-991f928f379e","Type":"ContainerStarted","Data":"aebb2ba810ac682ca863c77d594a4b3dc6a610a28c1a1d38a749157c2187840e"}
Nov 03 22:25:59.793131 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:59.793095    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh" event={"ID":"6855011d-1f9e-43be-8557-991f928f379e","Type":"ContainerStarted","Data":"7f1553beb8de03809ddb93d30a7990bdec51366a28caae39f791b8fa48e38fd6"}
Nov 03 22:25:59.864237 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:25:59.863816    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh" podStartSLOduration=1.863795155 podStartE2EDuration="1.863795155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:25:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:25:59.855826479 +0000 UTC m=+783.881682342" watchObservedRunningTime="2026-11-03 22:25:59.863795155 +0000 UTC m=+783.889651020"
Nov 03 22:26:00.605573 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:00.605533    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh"
Nov 03 22:26:00.607190 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:00.607157    4431 patch_prober.go:29] interesting pod/e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Nov 03 22:26:00.607511 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:00.607244    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh" podUID="6855011d-1f9e-43be-8557-991f928f379e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Nov 03 22:26:01.107433 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:01.107392    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:26:01.283422 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:26:01.283388    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:26:01.605710 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:01.605672    4431 patch_prober.go:29] interesting pod/e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Nov 03 22:26:01.606110 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:01.605737    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh" podUID="6855011d-1f9e-43be-8557-991f928f379e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Nov 03 22:26:02.131127 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:02.131098    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:26:02.605332 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:02.605289    4431 patch_prober.go:29] interesting pod/e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Nov 03 22:26:02.605520 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:02.605359    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh" podUID="6855011d-1f9e-43be-8557-991f928f379e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Nov 03 22:26:02.764609 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:02.764567    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:26:03.605778 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:03.605736    4431 patch_prober.go:29] interesting pod/e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Nov 03 22:26:03.605975 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:03.605805    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh" podUID="6855011d-1f9e-43be-8557-991f928f379e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Nov 03 22:26:03.954509 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:03.954469    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-2d619f/distinct-w6-2d619f-kserve-74dbc56695-tnvtb"]
Nov 03 22:26:03.965516 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:03.965474    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-2d619f/distinct-w6-2d619f-kserve-74dbc56695-tnvtb"
Nov 03 22:26:03.971873 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:03.971840    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-2d619f/distinct-w6-2d619f-kserve-74dbc56695-tnvtb"]
Nov 03 22:26:03.992273 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:03.992246    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-2d619f\"/\"default-dockercfg-krbfz\""
Nov 03 22:26:03.992446 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:03.992301    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-2d619f\"/\"distinct-w6-2d619f-kserve-self-signed-certs\""
Nov 03 22:26:03.992525 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:03.992399    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-2d619f\"/\"openshift-service-ca.crt\""
Nov 03 22:26:03.992525 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:03.992430    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-2d619f\"/\"kube-root-ca.crt\""
Nov 03 22:26:04.045068 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:04.045031    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90310d72-c818-4fdd-b4e7-b51f633caa05-home\") pod \"distinct-w6-2d619f-kserve-74dbc56695-tnvtb\" (UID: \"90310d72-c818-4fdd-b4e7-b51f633caa05\") " pod="e2e-models-e2e-worker-w6-2d619f/distinct-w6-2d619f-kserve-74dbc56695-tnvtb"
Nov 03 22:26:04.045431 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:04.045410    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90310d72-c818-4fdd-b4e7-b51f633caa05-dshm\") pod \"distinct-w6-2d619f-kserve-74dbc56695-tnvtb\" (UID: \"90310d72-c818-4fdd-b4e7-b51f633caa05\") " pod="e2e-models-e2e-worker-w6-2d619f/distinct-w6-2d619f-kserve-74dbc56695-tnvtb"
Nov 03 22:26:04.045699 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:04.045678    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90310d72-c818-4fdd-b4e7-b51f633caa05-model-cache\") pod \"distinct-w6-2d619f-kserve-74dbc56695-tnvtb\" (UID: \"90310d72-c818-4fdd-b4e7-b51f633caa05\") " pod="e2e-models-e2e-worker-w6-2d619f/distinct-w6-2d619f-kserve-74dbc56695-tnvtb"
Nov 03 22:26:04.045874 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:04.045859    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90310d72-c818-4fdd-b4e7-b51f633caa05-tmp-dir\") pod \"distinct-w6-2d619f-kserve-74dbc56695-tnvtb\" (UID: \"90310d72-c818-4fdd-b4e7-b51f633caa05\") " pod="e2e-models-e2e-worker-w6-2d619f/distinct-w6-2d619f-kserve-74dbc56695-tnvtb"
Nov 03 22:26:04.055446 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:04.055413    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90310d72-c818-4fdd-b4e7-b51f633caa05-tls-certs\") pod \"distinct-w6-2d619f-kserve-74dbc56695-tnvtb\" (UID: \"90310d72-c818-4fdd-b4e7-b51f633caa05\") " pod="e2e-models-e2e-worker-w6-2d619f/distinct-w6-2d619f-kserve-74dbc56695-tnvtb"
Nov 03 22:26:04.055664 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:04.055641    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvsks\" (UniqueName: \"kubernetes.io/projected/90310d72-c818-4fdd-b4e7-b51f633caa05-kube-api-access-pvsks\") pod \"distinct-w6-2d619f-kserve-74dbc56695-tnvtb\" (UID: \"90310d72-c818-4fdd-b4e7-b51f633caa05\") " pod="e2e-models-e2e-worker-w6-2d619f/distinct-w6-2d619f-kserve-74dbc56695-tnvtb"
Nov 03 22:26:04.059246 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:04.057019    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:26:04.160979 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:04.160935    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90310d72-c818-4fdd-b4e7-b51f633caa05-home\") pod \"distinct-w6-2d619f-kserve-74dbc56695-tnvtb\" (UID: \"90310d72-c818-4fdd-b4e7-b51f633caa05\") " pod="e2e-models-e2e-worker-w6-2d619f/distinct-w6-2d619f-kserve-74dbc56695-tnvtb"
Nov 03 22:26:04.162687 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:04.161241    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90310d72-c818-4fdd-b4e7-b51f633caa05-dshm\") pod \"distinct-w6-2d619f-kserve-74dbc56695-tnvtb\" (UID: \"90310d72-c818-4fdd-b4e7-b51f633caa05\") " pod="e2e-models-e2e-worker-w6-2d619f/distinct-w6-2d619f-kserve-74dbc56695-tnvtb"
Nov 03 22:26:04.162687 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:04.161361    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90310d72-c818-4fdd-b4e7-b51f633caa05-model-cache\") pod \"distinct-w6-2d619f-kserve-74dbc56695-tnvtb\" (UID: \"90310d72-c818-4fdd-b4e7-b51f633caa05\") " pod="e2e-models-e2e-worker-w6-2d619f/distinct-w6-2d619f-kserve-74dbc56695-tnvtb"
Nov 03 22:26:04.162687 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:04.161396    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90310d72-c818-4fdd-b4e7-b51f633caa05-tmp-dir\") pod \"distinct-w6-2d619f-kserve-74dbc56695-tnvtb\" (UID: \"90310d72-c818-4fdd-b4e7-b51f633caa05\") " pod="e2e-models-e2e-worker-w6-2d619f/distinct-w6-2d619f-kserve-74dbc56695-tnvtb"
Nov 03 22:26:04.162687 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:04.161524    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90310d72-c818-4fdd-b4e7-b51f633caa05-tls-certs\") pod \"distinct-w6-2d619f-kserve-74dbc56695-tnvtb\" (UID: \"90310d72-c818-4fdd-b4e7-b51f633caa05\") " pod="e2e-models-e2e-worker-w6-2d619f/distinct-w6-2d619f-kserve-74dbc56695-tnvtb"
Nov 03 22:26:04.162687 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:04.161560    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvsks\" (UniqueName: \"kubernetes.io/projected/90310d72-c818-4fdd-b4e7-b51f633caa05-kube-api-access-pvsks\") pod \"distinct-w6-2d619f-kserve-74dbc56695-tnvtb\" (UID: \"90310d72-c818-4fdd-b4e7-b51f633caa05\") " pod="e2e-models-e2e-worker-w6-2d619f/distinct-w6-2d619f-kserve-74dbc56695-tnvtb"
Nov 03 22:26:04.162687 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:04.162373    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90310d72-c818-4fdd-b4e7-b51f633caa05-home\") pod \"distinct-w6-2d619f-kserve-74dbc56695-tnvtb\" (UID: \"90310d72-c818-4fdd-b4e7-b51f633caa05\") " pod="e2e-models-e2e-worker-w6-2d619f/distinct-w6-2d619f-kserve-74dbc56695-tnvtb"
Nov 03 22:26:04.163407 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:04.163386    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90310d72-c818-4fdd-b4e7-b51f633caa05-tmp-dir\") pod \"distinct-w6-2d619f-kserve-74dbc56695-tnvtb\" (UID: \"90310d72-c818-4fdd-b4e7-b51f633caa05\") " pod="e2e-models-e2e-worker-w6-2d619f/distinct-w6-2d619f-kserve-74dbc56695-tnvtb"
Nov 03 22:26:04.164126 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:04.164079    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90310d72-c818-4fdd-b4e7-b51f633caa05-model-cache\") pod \"distinct-w6-2d619f-kserve-74dbc56695-tnvtb\" (UID: \"90310d72-c818-4fdd-b4e7-b51f633caa05\") " pod="e2e-models-e2e-worker-w6-2d619f/distinct-w6-2d619f-kserve-74dbc56695-tnvtb"
Nov 03 22:26:04.169696 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:04.169645    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90310d72-c818-4fdd-b4e7-b51f633caa05-dshm\") pod \"distinct-w6-2d619f-kserve-74dbc56695-tnvtb\" (UID: \"90310d72-c818-4fdd-b4e7-b51f633caa05\") " pod="e2e-models-e2e-worker-w6-2d619f/distinct-w6-2d619f-kserve-74dbc56695-tnvtb"
Nov 03 22:26:04.180171 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:04.180143    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90310d72-c818-4fdd-b4e7-b51f633caa05-tls-certs\") pod \"distinct-w6-2d619f-kserve-74dbc56695-tnvtb\" (UID: \"90310d72-c818-4fdd-b4e7-b51f633caa05\") " pod="e2e-models-e2e-worker-w6-2d619f/distinct-w6-2d619f-kserve-74dbc56695-tnvtb"
Nov 03 22:26:04.193089 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:04.193056    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvsks\" (UniqueName: \"kubernetes.io/projected/90310d72-c818-4fdd-b4e7-b51f633caa05-kube-api-access-pvsks\") pod \"distinct-w6-2d619f-kserve-74dbc56695-tnvtb\" (UID: \"90310d72-c818-4fdd-b4e7-b51f633caa05\") " pod="e2e-models-e2e-worker-w6-2d619f/distinct-w6-2d619f-kserve-74dbc56695-tnvtb"
Nov 03 22:26:04.284631 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:04.284540    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-2d619f/distinct-w6-2d619f-kserve-74dbc56695-tnvtb"
Nov 03 22:26:04.388127 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:04.387989    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:26:04.605640 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:04.605567    4431 patch_prober.go:29] interesting pod/e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Nov 03 22:26:04.605640 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:04.605617    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh" podUID="6855011d-1f9e-43be-8557-991f928f379e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Nov 03 22:26:04.613093 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:04.613067    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-2d619f/distinct-w6-2d619f-kserve-74dbc56695-tnvtb"]
Nov 03 22:26:04.822534 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:04.822497    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-2d619f/distinct-w6-2d619f-kserve-74dbc56695-tnvtb" event={"ID":"90310d72-c818-4fdd-b4e7-b51f633caa05","Type":"ContainerStarted","Data":"9dd05235d00df0d2082578e750a3ee62af070edd7e416545b7ef7d4f2ec94c6d"}
Nov 03 22:26:04.822722 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:04.822542    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-2d619f/distinct-w6-2d619f-kserve-74dbc56695-tnvtb" event={"ID":"90310d72-c818-4fdd-b4e7-b51f633caa05","Type":"ContainerStarted","Data":"c11b90afa48515845b5bef4382d60f8d16058e528e9de82e51ec2d5d31a3dbc9"}
Nov 03 22:26:04.864338 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:04.864206    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-2d619f/distinct-w6-2d619f-kserve-74dbc56695-tnvtb" podStartSLOduration=1.86418552 podStartE2EDuration="1.86418552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:26:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:26:04.862471201 +0000 UTC m=+788.888327058" watchObservedRunningTime="2026-11-03 22:26:04.86418552 +0000 UTC m=+788.890041386"
Nov 03 22:26:05.285719 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:05.285684    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2d619f/distinct-w6-2d619f-kserve-74dbc56695-tnvtb"
Nov 03 22:26:05.618503 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:05.618425    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh"
Nov 03 22:26:05.618859 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:05.618832    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh"
Nov 03 22:26:05.619698 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:05.619677    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh"
Nov 03 22:26:05.913504 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:05.913470    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:26:07.612625 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:07.612586    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:26:09.106125 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:09.106089    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:26:10.478157 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:10.478119    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:26:10.708544 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:10.708509    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:26:12.495843 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:12.495807    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:26:14.111648 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:14.111611    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:26:14.285040 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:14.285004    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-2d619f/distinct-w6-2d619f-kserve-74dbc56695-tnvtb"
Nov 03 22:26:14.297930 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:14.297901    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-2d619f/distinct-w6-2d619f-kserve-74dbc56695-tnvtb"
Nov 03 22:26:14.904056 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:14.904026    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2d619f/distinct-w6-2d619f-kserve-74dbc56695-tnvtb"
Nov 03 22:26:15.749073 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:15.749035    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:26:17.374554 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:17.374501    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:26:17.531382 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:17.531249    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:26:19.225009 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:19.224967    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:26:20.820888 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:20.820856    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:26:22.310029 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:22.309995    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:26:23.806426 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:23.806390    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:26:24.381633 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:24.381593    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-50d738/distinct-w4-50d738-kserve-668fdf8b86-5pp9p"]
Nov 03 22:26:24.386611 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:24.386583    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-50d738/distinct-w4-50d738-kserve-668fdf8b86-5pp9p"
Nov 03 22:26:24.402625 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:24.402595    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-50d738/distinct-w4-50d738-kserve-668fdf8b86-5pp9p"]
Nov 03 22:26:24.411529 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:24.411503    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-50d738\"/\"distinct-w4-50d738-kserve-self-signed-certs\""
Nov 03 22:26:24.442136 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:24.442101    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce50af62-9cb6-4379-99c7-b40630538f4f-home\") pod \"distinct-w4-50d738-kserve-668fdf8b86-5pp9p\" (UID: \"ce50af62-9cb6-4379-99c7-b40630538f4f\") " pod="e2e-models-e2e-worker-w4-50d738/distinct-w4-50d738-kserve-668fdf8b86-5pp9p"
Nov 03 22:26:24.442308 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:24.442254    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce50af62-9cb6-4379-99c7-b40630538f4f-dshm\") pod \"distinct-w4-50d738-kserve-668fdf8b86-5pp9p\" (UID: \"ce50af62-9cb6-4379-99c7-b40630538f4f\") " pod="e2e-models-e2e-worker-w4-50d738/distinct-w4-50d738-kserve-668fdf8b86-5pp9p"
Nov 03 22:26:24.442308 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:24.442293    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k98d\" (UniqueName: \"kubernetes.io/projected/ce50af62-9cb6-4379-99c7-b40630538f4f-kube-api-access-8k98d\") pod \"distinct-w4-50d738-kserve-668fdf8b86-5pp9p\" (UID: \"ce50af62-9cb6-4379-99c7-b40630538f4f\") " pod="e2e-models-e2e-worker-w4-50d738/distinct-w4-50d738-kserve-668fdf8b86-5pp9p"
Nov 03 22:26:24.442408 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:24.442358    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce50af62-9cb6-4379-99c7-b40630538f4f-tls-certs\") pod \"distinct-w4-50d738-kserve-668fdf8b86-5pp9p\" (UID: \"ce50af62-9cb6-4379-99c7-b40630538f4f\") " pod="e2e-models-e2e-worker-w4-50d738/distinct-w4-50d738-kserve-668fdf8b86-5pp9p"
Nov 03 22:26:24.442408 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:24.442378    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce50af62-9cb6-4379-99c7-b40630538f4f-tmp-dir\") pod \"distinct-w4-50d738-kserve-668fdf8b86-5pp9p\" (UID: \"ce50af62-9cb6-4379-99c7-b40630538f4f\") " pod="e2e-models-e2e-worker-w4-50d738/distinct-w4-50d738-kserve-668fdf8b86-5pp9p"
Nov 03 22:26:24.442408 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:24.442403    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce50af62-9cb6-4379-99c7-b40630538f4f-model-cache\") pod \"distinct-w4-50d738-kserve-668fdf8b86-5pp9p\" (UID: \"ce50af62-9cb6-4379-99c7-b40630538f4f\") " pod="e2e-models-e2e-worker-w4-50d738/distinct-w4-50d738-kserve-668fdf8b86-5pp9p"
Nov 03 22:26:24.471320 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:24.471265    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:26:24.543358 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:24.543323    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce50af62-9cb6-4379-99c7-b40630538f4f-dshm\") pod \"distinct-w4-50d738-kserve-668fdf8b86-5pp9p\" (UID: \"ce50af62-9cb6-4379-99c7-b40630538f4f\") " pod="e2e-models-e2e-worker-w4-50d738/distinct-w4-50d738-kserve-668fdf8b86-5pp9p"
Nov 03 22:26:24.543358 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:24.543360    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k98d\" (UniqueName: \"kubernetes.io/projected/ce50af62-9cb6-4379-99c7-b40630538f4f-kube-api-access-8k98d\") pod \"distinct-w4-50d738-kserve-668fdf8b86-5pp9p\" (UID: \"ce50af62-9cb6-4379-99c7-b40630538f4f\") " pod="e2e-models-e2e-worker-w4-50d738/distinct-w4-50d738-kserve-668fdf8b86-5pp9p"
Nov 03 22:26:24.543618 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:24.543423    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce50af62-9cb6-4379-99c7-b40630538f4f-tls-certs\") pod \"distinct-w4-50d738-kserve-668fdf8b86-5pp9p\" (UID: \"ce50af62-9cb6-4379-99c7-b40630538f4f\") " pod="e2e-models-e2e-worker-w4-50d738/distinct-w4-50d738-kserve-668fdf8b86-5pp9p"
Nov 03 22:26:24.543618 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:24.543441    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce50af62-9cb6-4379-99c7-b40630538f4f-tmp-dir\") pod \"distinct-w4-50d738-kserve-668fdf8b86-5pp9p\" (UID: \"ce50af62-9cb6-4379-99c7-b40630538f4f\") " pod="e2e-models-e2e-worker-w4-50d738/distinct-w4-50d738-kserve-668fdf8b86-5pp9p"
Nov 03 22:26:24.543618 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:24.543464    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce50af62-9cb6-4379-99c7-b40630538f4f-model-cache\") pod \"distinct-w4-50d738-kserve-668fdf8b86-5pp9p\" (UID: \"ce50af62-9cb6-4379-99c7-b40630538f4f\") " pod="e2e-models-e2e-worker-w4-50d738/distinct-w4-50d738-kserve-668fdf8b86-5pp9p"
Nov 03 22:26:24.543618 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:24.543575    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce50af62-9cb6-4379-99c7-b40630538f4f-home\") pod \"distinct-w4-50d738-kserve-668fdf8b86-5pp9p\" (UID: \"ce50af62-9cb6-4379-99c7-b40630538f4f\") " pod="e2e-models-e2e-worker-w4-50d738/distinct-w4-50d738-kserve-668fdf8b86-5pp9p"
Nov 03 22:26:24.543982 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:24.543958    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce50af62-9cb6-4379-99c7-b40630538f4f-tmp-dir\") pod \"distinct-w4-50d738-kserve-668fdf8b86-5pp9p\" (UID: \"ce50af62-9cb6-4379-99c7-b40630538f4f\") " pod="e2e-models-e2e-worker-w4-50d738/distinct-w4-50d738-kserve-668fdf8b86-5pp9p"
Nov 03 22:26:24.544084 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:24.544020    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce50af62-9cb6-4379-99c7-b40630538f4f-model-cache\") pod \"distinct-w4-50d738-kserve-668fdf8b86-5pp9p\" (UID: \"ce50af62-9cb6-4379-99c7-b40630538f4f\") " pod="e2e-models-e2e-worker-w4-50d738/distinct-w4-50d738-kserve-668fdf8b86-5pp9p"
Nov 03 22:26:24.544125 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:24.544085    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce50af62-9cb6-4379-99c7-b40630538f4f-home\") pod \"distinct-w4-50d738-kserve-668fdf8b86-5pp9p\" (UID: \"ce50af62-9cb6-4379-99c7-b40630538f4f\") " pod="e2e-models-e2e-worker-w4-50d738/distinct-w4-50d738-kserve-668fdf8b86-5pp9p"
Nov 03 22:26:24.546243 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:24.546198    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce50af62-9cb6-4379-99c7-b40630538f4f-dshm\") pod \"distinct-w4-50d738-kserve-668fdf8b86-5pp9p\" (UID: \"ce50af62-9cb6-4379-99c7-b40630538f4f\") " pod="e2e-models-e2e-worker-w4-50d738/distinct-w4-50d738-kserve-668fdf8b86-5pp9p"
Nov 03 22:26:24.546375 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:24.546354    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce50af62-9cb6-4379-99c7-b40630538f4f-tls-certs\") pod \"distinct-w4-50d738-kserve-668fdf8b86-5pp9p\" (UID: \"ce50af62-9cb6-4379-99c7-b40630538f4f\") " pod="e2e-models-e2e-worker-w4-50d738/distinct-w4-50d738-kserve-668fdf8b86-5pp9p"
Nov 03 22:26:24.577963 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:24.577926    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k98d\" (UniqueName: \"kubernetes.io/projected/ce50af62-9cb6-4379-99c7-b40630538f4f-kube-api-access-8k98d\") pod \"distinct-w4-50d738-kserve-668fdf8b86-5pp9p\" (UID: \"ce50af62-9cb6-4379-99c7-b40630538f4f\") " pod="e2e-models-e2e-worker-w4-50d738/distinct-w4-50d738-kserve-668fdf8b86-5pp9p"
Nov 03 22:26:24.703984 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:24.703944    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-50d738/distinct-w4-50d738-kserve-668fdf8b86-5pp9p"
Nov 03 22:26:25.010467 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:25.010443    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-50d738/distinct-w4-50d738-kserve-668fdf8b86-5pp9p"]
Nov 03 22:26:25.012181 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:26:25.012154    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podce50af62_9cb6_4379_99c7_b40630538f4f.slice/crio-af9703ee1a26d47150e2375316b7419e5c2f2a28add50fdd7938bdcdad9a670e WatchSource:0}: Error finding container af9703ee1a26d47150e2375316b7419e5c2f2a28add50fdd7938bdcdad9a670e: Status 404 returned error can't find the container with id af9703ee1a26d47150e2375316b7419e5c2f2a28add50fdd7938bdcdad9a670e
Nov 03 22:26:25.321554 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:25.321456    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:26:25.956858 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:25.956825    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-50d738/distinct-w4-50d738-kserve-668fdf8b86-5pp9p" event={"ID":"ce50af62-9cb6-4379-99c7-b40630538f4f","Type":"ContainerStarted","Data":"34d313d7289e50e19bdb13e70d2265b439795cfc19fa1ac6a29ecc5280fa61c3"}
Nov 03 22:26:25.956858 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:25.956864    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-50d738/distinct-w4-50d738-kserve-668fdf8b86-5pp9p" event={"ID":"ce50af62-9cb6-4379-99c7-b40630538f4f","Type":"ContainerStarted","Data":"af9703ee1a26d47150e2375316b7419e5c2f2a28add50fdd7938bdcdad9a670e"}
Nov 03 22:26:25.999551 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:25.999497    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-50d738/distinct-w4-50d738-kserve-668fdf8b86-5pp9p" podStartSLOduration=1.999482822 podStartE2EDuration="1.999482822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:26:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:26:25.997088854 +0000 UTC m=+810.022944730" watchObservedRunningTime="2026-11-03 22:26:25.999482822 +0000 UTC m=+810.025338683"
Nov 03 22:26:26.704614 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:26.704574    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-50d738/distinct-w4-50d738-kserve-668fdf8b86-5pp9p"
Nov 03 22:26:26.821725 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:26.821690    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:26:27.111093 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:27.111011    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:26:28.448973 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:28.448891    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:26:30.134298 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:30.134257    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:26:31.258629 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:26:31.258592    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:26:31.461433 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:31.461390    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-5187e1/distinct-w3-5187e1-kserve-545b7fb6bc-mk49g"]
Nov 03 22:26:31.466439 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:31.466411    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5187e1/distinct-w3-5187e1-kserve-545b7fb6bc-mk49g"
Nov 03 22:26:31.483907 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:31.483879    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-5187e1/distinct-w3-5187e1-kserve-545b7fb6bc-mk49g"]
Nov 03 22:26:31.491609 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:31.491581    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-5187e1\"/\"distinct-w3-5187e1-kserve-self-signed-certs\""
Nov 03 22:26:31.548915 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:31.548810    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:26:31.632652 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:31.632619    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ba8dc17-1b1b-4811-9dd6-595d75c6ca04-tmp-dir\") pod \"distinct-w3-5187e1-kserve-545b7fb6bc-mk49g\" (UID: \"3ba8dc17-1b1b-4811-9dd6-595d75c6ca04\") " pod="e2e-models-e2e-worker-w3-5187e1/distinct-w3-5187e1-kserve-545b7fb6bc-mk49g"
Nov 03 22:26:31.632825 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:31.632698    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ba8dc17-1b1b-4811-9dd6-595d75c6ca04-tls-certs\") pod \"distinct-w3-5187e1-kserve-545b7fb6bc-mk49g\" (UID: \"3ba8dc17-1b1b-4811-9dd6-595d75c6ca04\") " pod="e2e-models-e2e-worker-w3-5187e1/distinct-w3-5187e1-kserve-545b7fb6bc-mk49g"
Nov 03 22:26:31.632825 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:31.632784    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ba8dc17-1b1b-4811-9dd6-595d75c6ca04-home\") pod \"distinct-w3-5187e1-kserve-545b7fb6bc-mk49g\" (UID: \"3ba8dc17-1b1b-4811-9dd6-595d75c6ca04\") " pod="e2e-models-e2e-worker-w3-5187e1/distinct-w3-5187e1-kserve-545b7fb6bc-mk49g"
Nov 03 22:26:31.632899 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:31.632857    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5qbj\" (UniqueName: \"kubernetes.io/projected/3ba8dc17-1b1b-4811-9dd6-595d75c6ca04-kube-api-access-p5qbj\") pod \"distinct-w3-5187e1-kserve-545b7fb6bc-mk49g\" (UID: \"3ba8dc17-1b1b-4811-9dd6-595d75c6ca04\") " pod="e2e-models-e2e-worker-w3-5187e1/distinct-w3-5187e1-kserve-545b7fb6bc-mk49g"
Nov 03 22:26:31.632996 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:31.632968    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ba8dc17-1b1b-4811-9dd6-595d75c6ca04-model-cache\") pod \"distinct-w3-5187e1-kserve-545b7fb6bc-mk49g\" (UID: \"3ba8dc17-1b1b-4811-9dd6-595d75c6ca04\") " pod="e2e-models-e2e-worker-w3-5187e1/distinct-w3-5187e1-kserve-545b7fb6bc-mk49g"
Nov 03 22:26:31.633045 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:31.633029    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ba8dc17-1b1b-4811-9dd6-595d75c6ca04-dshm\") pod \"distinct-w3-5187e1-kserve-545b7fb6bc-mk49g\" (UID: \"3ba8dc17-1b1b-4811-9dd6-595d75c6ca04\") " pod="e2e-models-e2e-worker-w3-5187e1/distinct-w3-5187e1-kserve-545b7fb6bc-mk49g"
Nov 03 22:26:31.734470 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:31.734430    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ba8dc17-1b1b-4811-9dd6-595d75c6ca04-model-cache\") pod \"distinct-w3-5187e1-kserve-545b7fb6bc-mk49g\" (UID: \"3ba8dc17-1b1b-4811-9dd6-595d75c6ca04\") " pod="e2e-models-e2e-worker-w3-5187e1/distinct-w3-5187e1-kserve-545b7fb6bc-mk49g"
Nov 03 22:26:31.734675 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:31.734501    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ba8dc17-1b1b-4811-9dd6-595d75c6ca04-dshm\") pod \"distinct-w3-5187e1-kserve-545b7fb6bc-mk49g\" (UID: \"3ba8dc17-1b1b-4811-9dd6-595d75c6ca04\") " pod="e2e-models-e2e-worker-w3-5187e1/distinct-w3-5187e1-kserve-545b7fb6bc-mk49g"
Nov 03 22:26:31.734675 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:31.734553    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ba8dc17-1b1b-4811-9dd6-595d75c6ca04-tmp-dir\") pod \"distinct-w3-5187e1-kserve-545b7fb6bc-mk49g\" (UID: \"3ba8dc17-1b1b-4811-9dd6-595d75c6ca04\") " pod="e2e-models-e2e-worker-w3-5187e1/distinct-w3-5187e1-kserve-545b7fb6bc-mk49g"
Nov 03 22:26:31.734808 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:31.734683    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ba8dc17-1b1b-4811-9dd6-595d75c6ca04-tls-certs\") pod \"distinct-w3-5187e1-kserve-545b7fb6bc-mk49g\" (UID: \"3ba8dc17-1b1b-4811-9dd6-595d75c6ca04\") " pod="e2e-models-e2e-worker-w3-5187e1/distinct-w3-5187e1-kserve-545b7fb6bc-mk49g"
Nov 03 22:26:31.734808 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:31.734732    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ba8dc17-1b1b-4811-9dd6-595d75c6ca04-home\") pod \"distinct-w3-5187e1-kserve-545b7fb6bc-mk49g\" (UID: \"3ba8dc17-1b1b-4811-9dd6-595d75c6ca04\") " pod="e2e-models-e2e-worker-w3-5187e1/distinct-w3-5187e1-kserve-545b7fb6bc-mk49g"
Nov 03 22:26:31.734910 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:31.734856    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5qbj\" (UniqueName: \"kubernetes.io/projected/3ba8dc17-1b1b-4811-9dd6-595d75c6ca04-kube-api-access-p5qbj\") pod \"distinct-w3-5187e1-kserve-545b7fb6bc-mk49g\" (UID: \"3ba8dc17-1b1b-4811-9dd6-595d75c6ca04\") " pod="e2e-models-e2e-worker-w3-5187e1/distinct-w3-5187e1-kserve-545b7fb6bc-mk49g"
Nov 03 22:26:31.735357 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:31.734955    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ba8dc17-1b1b-4811-9dd6-595d75c6ca04-model-cache\") pod \"distinct-w3-5187e1-kserve-545b7fb6bc-mk49g\" (UID: \"3ba8dc17-1b1b-4811-9dd6-595d75c6ca04\") " pod="e2e-models-e2e-worker-w3-5187e1/distinct-w3-5187e1-kserve-545b7fb6bc-mk49g"
Nov 03 22:26:31.735357 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:31.734980    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ba8dc17-1b1b-4811-9dd6-595d75c6ca04-tmp-dir\") pod \"distinct-w3-5187e1-kserve-545b7fb6bc-mk49g\" (UID: \"3ba8dc17-1b1b-4811-9dd6-595d75c6ca04\") " pod="e2e-models-e2e-worker-w3-5187e1/distinct-w3-5187e1-kserve-545b7fb6bc-mk49g"
Nov 03 22:26:31.735357 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:31.735312    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ba8dc17-1b1b-4811-9dd6-595d75c6ca04-home\") pod \"distinct-w3-5187e1-kserve-545b7fb6bc-mk49g\" (UID: \"3ba8dc17-1b1b-4811-9dd6-595d75c6ca04\") " pod="e2e-models-e2e-worker-w3-5187e1/distinct-w3-5187e1-kserve-545b7fb6bc-mk49g"
Nov 03 22:26:31.736943 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:31.736913    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ba8dc17-1b1b-4811-9dd6-595d75c6ca04-dshm\") pod \"distinct-w3-5187e1-kserve-545b7fb6bc-mk49g\" (UID: \"3ba8dc17-1b1b-4811-9dd6-595d75c6ca04\") " pod="e2e-models-e2e-worker-w3-5187e1/distinct-w3-5187e1-kserve-545b7fb6bc-mk49g"
Nov 03 22:26:31.737245 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:31.737194    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ba8dc17-1b1b-4811-9dd6-595d75c6ca04-tls-certs\") pod \"distinct-w3-5187e1-kserve-545b7fb6bc-mk49g\" (UID: \"3ba8dc17-1b1b-4811-9dd6-595d75c6ca04\") " pod="e2e-models-e2e-worker-w3-5187e1/distinct-w3-5187e1-kserve-545b7fb6bc-mk49g"
Nov 03 22:26:31.758575 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:31.758542    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5qbj\" (UniqueName: \"kubernetes.io/projected/3ba8dc17-1b1b-4811-9dd6-595d75c6ca04-kube-api-access-p5qbj\") pod \"distinct-w3-5187e1-kserve-545b7fb6bc-mk49g\" (UID: \"3ba8dc17-1b1b-4811-9dd6-595d75c6ca04\") " pod="e2e-models-e2e-worker-w3-5187e1/distinct-w3-5187e1-kserve-545b7fb6bc-mk49g"
Nov 03 22:26:31.783043 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:31.783009    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5187e1/distinct-w3-5187e1-kserve-545b7fb6bc-mk49g"
Nov 03 22:26:31.926246 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:31.925132    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:26:32.091706 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:32.091646    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-5187e1/distinct-w3-5187e1-kserve-545b7fb6bc-mk49g"]
Nov 03 22:26:32.093366 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:26:32.093342    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3ba8dc17_1b1b_4811_9dd6_595d75c6ca04.slice/crio-259a55885b9da9a48c94bfe08702e73a9112980e01298ef511d51eb3aa9e3cde WatchSource:0}: Error finding container 259a55885b9da9a48c94bfe08702e73a9112980e01298ef511d51eb3aa9e3cde: Status 404 returned error can't find the container with id 259a55885b9da9a48c94bfe08702e73a9112980e01298ef511d51eb3aa9e3cde
Nov 03 22:26:32.997204 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:32.997167    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5187e1/distinct-w3-5187e1-kserve-545b7fb6bc-mk49g" event={"ID":"3ba8dc17-1b1b-4811-9dd6-595d75c6ca04","Type":"ContainerStarted","Data":"d9e2c5542308e597a66991f01ebfca061a9973a8ecae78e378eb21aa4f5e005e"}
Nov 03 22:26:32.997627 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:32.997233    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5187e1/distinct-w3-5187e1-kserve-545b7fb6bc-mk49g" event={"ID":"3ba8dc17-1b1b-4811-9dd6-595d75c6ca04","Type":"ContainerStarted","Data":"259a55885b9da9a48c94bfe08702e73a9112980e01298ef511d51eb3aa9e3cde"}
Nov 03 22:26:33.039030 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:33.038976    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-5187e1/distinct-w3-5187e1-kserve-545b7fb6bc-mk49g" podStartSLOduration=2.038962347 podStartE2EDuration="2.038962347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:26:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:26:33.037165971 +0000 UTC m=+817.063021834" watchObservedRunningTime="2026-11-03 22:26:33.038962347 +0000 UTC m=+817.064818209"
Nov 03 22:26:33.564012 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:33.563956    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:26:33.783947 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:33.783905    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5187e1/distinct-w3-5187e1-kserve-545b7fb6bc-mk49g"
Nov 03 22:26:34.704142 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:34.704106    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-50d738/distinct-w4-50d738-kserve-668fdf8b86-5pp9p"
Nov 03 22:26:34.717159 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:34.717133    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-50d738/distinct-w4-50d738-kserve-668fdf8b86-5pp9p"
Nov 03 22:26:35.019316 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:35.019237    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-50d738/distinct-w4-50d738-kserve-668fdf8b86-5pp9p"
Nov 03 22:26:35.241573 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:35.241528    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:26:36.901621 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:36.901588    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:26:38.472203 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:38.472151    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:26:40.100298 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:40.100226    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:26:40.768433 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:40.768400    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:26:41.776949 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:41.776910    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:26:41.783948 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:41.783920    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-5187e1/distinct-w3-5187e1-kserve-545b7fb6bc-mk49g"
Nov 03 22:26:41.799497 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:41.799464    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-5187e1/distinct-w3-5187e1-kserve-545b7fb6bc-mk49g"
Nov 03 22:26:42.075528 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:42.075418    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5187e1/distinct-w3-5187e1-kserve-545b7fb6bc-mk49g"
Nov 03 22:26:43.467584 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:43.467548    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:26:45.033677 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:45.033634    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:26:45.597510 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:45.597477    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:26:46.618864 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:46.618826    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:26:48.315116 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:48.315083    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:26:49.818522 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:49.818491    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:26:50.363361 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:50.363321    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:26:51.499230 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:51.499188    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:26:53.092518 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:53.092486    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:26:54.703859 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:54.703821    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:26:56.214431 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:56.214400    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:26:57.811463 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:57.811427    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:26:59.500829 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:26:59.500791    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:27:01.253134 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:01.253081    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:27:01.268113 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:27:01.268069    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:27:01.884687 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:01.884654    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-553753/distinct-2-w2-553753-kserve-5bf8568844-7q4l2"]
Nov 03 22:27:01.890119 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:01.890089    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-553753/distinct-2-w2-553753-kserve-5bf8568844-7q4l2"]
Nov 03 22:27:01.890304 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:01.890177    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-553753/distinct-2-w2-553753-kserve-5bf8568844-7q4l2"
Nov 03 22:27:01.897964 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:01.897919    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1894b80-164c-4ce1-94f3-1e5011d9759e-home\") pod \"distinct-2-w2-553753-kserve-5bf8568844-7q4l2\" (UID: \"a1894b80-164c-4ce1-94f3-1e5011d9759e\") " pod="e2e-models-e2e-worker-w2-553753/distinct-2-w2-553753-kserve-5bf8568844-7q4l2"
Nov 03 22:27:01.898264 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:01.898230    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1894b80-164c-4ce1-94f3-1e5011d9759e-dshm\") pod \"distinct-2-w2-553753-kserve-5bf8568844-7q4l2\" (UID: \"a1894b80-164c-4ce1-94f3-1e5011d9759e\") " pod="e2e-models-e2e-worker-w2-553753/distinct-2-w2-553753-kserve-5bf8568844-7q4l2"
Nov 03 22:27:01.898493 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:01.898466    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwxwf\" (UniqueName: \"kubernetes.io/projected/a1894b80-164c-4ce1-94f3-1e5011d9759e-kube-api-access-kwxwf\") pod \"distinct-2-w2-553753-kserve-5bf8568844-7q4l2\" (UID: \"a1894b80-164c-4ce1-94f3-1e5011d9759e\") " pod="e2e-models-e2e-worker-w2-553753/distinct-2-w2-553753-kserve-5bf8568844-7q4l2"
Nov 03 22:27:01.898664 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:01.898633    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1894b80-164c-4ce1-94f3-1e5011d9759e-tls-certs\") pod \"distinct-2-w2-553753-kserve-5bf8568844-7q4l2\" (UID: \"a1894b80-164c-4ce1-94f3-1e5011d9759e\") " pod="e2e-models-e2e-worker-w2-553753/distinct-2-w2-553753-kserve-5bf8568844-7q4l2"
Nov 03 22:27:01.898827 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:01.898808    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1894b80-164c-4ce1-94f3-1e5011d9759e-model-cache\") pod \"distinct-2-w2-553753-kserve-5bf8568844-7q4l2\" (UID: \"a1894b80-164c-4ce1-94f3-1e5011d9759e\") " pod="e2e-models-e2e-worker-w2-553753/distinct-2-w2-553753-kserve-5bf8568844-7q4l2"
Nov 03 22:27:01.899069 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:01.899041    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1894b80-164c-4ce1-94f3-1e5011d9759e-tmp-dir\") pod \"distinct-2-w2-553753-kserve-5bf8568844-7q4l2\" (UID: \"a1894b80-164c-4ce1-94f3-1e5011d9759e\") " pod="e2e-models-e2e-worker-w2-553753/distinct-2-w2-553753-kserve-5bf8568844-7q4l2"
Nov 03 22:27:01.916772 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:01.916749    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-553753\"/\"distinct-2-w2-553753-kserve-self-signed-certs\""
Nov 03 22:27:01.953922 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:01.953892    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:27:02.000550 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:02.000515    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1894b80-164c-4ce1-94f3-1e5011d9759e-tmp-dir\") pod \"distinct-2-w2-553753-kserve-5bf8568844-7q4l2\" (UID: \"a1894b80-164c-4ce1-94f3-1e5011d9759e\") " pod="e2e-models-e2e-worker-w2-553753/distinct-2-w2-553753-kserve-5bf8568844-7q4l2"
Nov 03 22:27:02.000756 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:02.000585    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1894b80-164c-4ce1-94f3-1e5011d9759e-home\") pod \"distinct-2-w2-553753-kserve-5bf8568844-7q4l2\" (UID: \"a1894b80-164c-4ce1-94f3-1e5011d9759e\") " pod="e2e-models-e2e-worker-w2-553753/distinct-2-w2-553753-kserve-5bf8568844-7q4l2"
Nov 03 22:27:02.000756 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:02.000647    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1894b80-164c-4ce1-94f3-1e5011d9759e-dshm\") pod \"distinct-2-w2-553753-kserve-5bf8568844-7q4l2\" (UID: \"a1894b80-164c-4ce1-94f3-1e5011d9759e\") " pod="e2e-models-e2e-worker-w2-553753/distinct-2-w2-553753-kserve-5bf8568844-7q4l2"
Nov 03 22:27:02.000756 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:02.000688    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwxwf\" (UniqueName: \"kubernetes.io/projected/a1894b80-164c-4ce1-94f3-1e5011d9759e-kube-api-access-kwxwf\") pod \"distinct-2-w2-553753-kserve-5bf8568844-7q4l2\" (UID: \"a1894b80-164c-4ce1-94f3-1e5011d9759e\") " pod="e2e-models-e2e-worker-w2-553753/distinct-2-w2-553753-kserve-5bf8568844-7q4l2"
Nov 03 22:27:02.000756 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:02.000718    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1894b80-164c-4ce1-94f3-1e5011d9759e-tls-certs\") pod \"distinct-2-w2-553753-kserve-5bf8568844-7q4l2\" (UID: \"a1894b80-164c-4ce1-94f3-1e5011d9759e\") " pod="e2e-models-e2e-worker-w2-553753/distinct-2-w2-553753-kserve-5bf8568844-7q4l2"
Nov 03 22:27:02.000756 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:02.000745    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1894b80-164c-4ce1-94f3-1e5011d9759e-model-cache\") pod \"distinct-2-w2-553753-kserve-5bf8568844-7q4l2\" (UID: \"a1894b80-164c-4ce1-94f3-1e5011d9759e\") " pod="e2e-models-e2e-worker-w2-553753/distinct-2-w2-553753-kserve-5bf8568844-7q4l2"
Nov 03 22:27:02.001038 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:02.000969    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1894b80-164c-4ce1-94f3-1e5011d9759e-tmp-dir\") pod \"distinct-2-w2-553753-kserve-5bf8568844-7q4l2\" (UID: \"a1894b80-164c-4ce1-94f3-1e5011d9759e\") " pod="e2e-models-e2e-worker-w2-553753/distinct-2-w2-553753-kserve-5bf8568844-7q4l2"
Nov 03 22:27:02.001175 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:02.001154    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1894b80-164c-4ce1-94f3-1e5011d9759e-model-cache\") pod \"distinct-2-w2-553753-kserve-5bf8568844-7q4l2\" (UID: \"a1894b80-164c-4ce1-94f3-1e5011d9759e\") " pod="e2e-models-e2e-worker-w2-553753/distinct-2-w2-553753-kserve-5bf8568844-7q4l2"
Nov 03 22:27:02.001377 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:02.001355    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1894b80-164c-4ce1-94f3-1e5011d9759e-home\") pod \"distinct-2-w2-553753-kserve-5bf8568844-7q4l2\" (UID: \"a1894b80-164c-4ce1-94f3-1e5011d9759e\") " pod="e2e-models-e2e-worker-w2-553753/distinct-2-w2-553753-kserve-5bf8568844-7q4l2"
Nov 03 22:27:02.003111 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:02.003083    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1894b80-164c-4ce1-94f3-1e5011d9759e-dshm\") pod \"distinct-2-w2-553753-kserve-5bf8568844-7q4l2\" (UID: \"a1894b80-164c-4ce1-94f3-1e5011d9759e\") " pod="e2e-models-e2e-worker-w2-553753/distinct-2-w2-553753-kserve-5bf8568844-7q4l2"
Nov 03 22:27:02.003413 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:02.003390    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1894b80-164c-4ce1-94f3-1e5011d9759e-tls-certs\") pod \"distinct-2-w2-553753-kserve-5bf8568844-7q4l2\" (UID: \"a1894b80-164c-4ce1-94f3-1e5011d9759e\") " pod="e2e-models-e2e-worker-w2-553753/distinct-2-w2-553753-kserve-5bf8568844-7q4l2"
Nov 03 22:27:02.025741 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:02.025713    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwxwf\" (UniqueName: \"kubernetes.io/projected/a1894b80-164c-4ce1-94f3-1e5011d9759e-kube-api-access-kwxwf\") pod \"distinct-2-w2-553753-kserve-5bf8568844-7q4l2\" (UID: \"a1894b80-164c-4ce1-94f3-1e5011d9759e\") " pod="e2e-models-e2e-worker-w2-553753/distinct-2-w2-553753-kserve-5bf8568844-7q4l2"
Nov 03 22:27:02.206695 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:02.206651    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-553753/distinct-2-w2-553753-kserve-5bf8568844-7q4l2"
Nov 03 22:27:02.534741 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:02.534706    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-553753/distinct-2-w2-553753-kserve-5bf8568844-7q4l2"]
Nov 03 22:27:02.536060 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:27:02.536031    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda1894b80_164c_4ce1_94f3_1e5011d9759e.slice/crio-9f2a9116560e76b9273d350f5a500cb7f7c5df22ffe572bacfd1be7fcd4eaf50 WatchSource:0}: Error finding container 9f2a9116560e76b9273d350f5a500cb7f7c5df22ffe572bacfd1be7fcd4eaf50: Status 404 returned error can't find the container with id 9f2a9116560e76b9273d350f5a500cb7f7c5df22ffe572bacfd1be7fcd4eaf50
Nov 03 22:27:03.178820 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:03.178786    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-553753/distinct-2-w2-553753-kserve-5bf8568844-7q4l2" event={"ID":"a1894b80-164c-4ce1-94f3-1e5011d9759e","Type":"ContainerStarted","Data":"bf6a6fbd1cd9ef11331329778a45ce7bb8ed898bc7ed45b815d4a45bcd5118e5"}
Nov 03 22:27:03.178998 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:03.178829    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-553753/distinct-2-w2-553753-kserve-5bf8568844-7q4l2" event={"ID":"a1894b80-164c-4ce1-94f3-1e5011d9759e","Type":"ContainerStarted","Data":"9f2a9116560e76b9273d350f5a500cb7f7c5df22ffe572bacfd1be7fcd4eaf50"}
Nov 03 22:27:03.206949 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:03.206919    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-553753/distinct-2-w2-553753-kserve-5bf8568844-7q4l2"
Nov 03 22:27:03.221086 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:03.221033    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-553753/distinct-2-w2-553753-kserve-5bf8568844-7q4l2" podStartSLOduration=2.221015424 podStartE2EDuration="2.221015424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:27:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:27:03.218961644 +0000 UTC m=+847.244817506" watchObservedRunningTime="2026-11-03 22:27:03.221015424 +0000 UTC m=+847.246871287"
Nov 03 22:27:03.335120 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:03.335085    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:27:05.045693 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:05.045656    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:27:06.796260 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:06.796227    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:27:08.407860 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:08.407826    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:27:10.003639 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:10.003579    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:27:11.620558 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:11.620517    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:27:12.207295 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:12.207260    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-553753/distinct-2-w2-553753-kserve-5bf8568844-7q4l2"
Nov 03 22:27:12.220317 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:12.220287    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-553753/distinct-2-w2-553753-kserve-5bf8568844-7q4l2"
Nov 03 22:27:12.240792 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:12.240758    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-553753/distinct-2-w2-553753-kserve-5bf8568844-7q4l2"
Nov 03 22:27:13.220492 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:13.220454    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:27:13.574972 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:13.574890    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:27:15.101570 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:15.101527    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:27:16.808393 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:16.808356    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:27:17.419048 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:17.419009    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-50d738/distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r"]
Nov 03 22:27:17.423754 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:17.423726    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-50d738/distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r"
Nov 03 22:27:17.441180 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:17.441150    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-50d738/distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r"]
Nov 03 22:27:17.448771 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:17.448743    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-50d738\"/\"distinct-2-w4-50d738-kserve-self-signed-certs\""
Nov 03 22:27:17.512197 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:17.512127    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:27:17.586139 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:17.586100    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e45b1572-6519-4c71-884a-115eeddd86cf-tmp-dir\") pod \"distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r\" (UID: \"e45b1572-6519-4c71-884a-115eeddd86cf\") " pod="e2e-models-e2e-worker-w4-50d738/distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r"
Nov 03 22:27:17.586335 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:17.586264    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-428n5\" (UniqueName: \"kubernetes.io/projected/e45b1572-6519-4c71-884a-115eeddd86cf-kube-api-access-428n5\") pod \"distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r\" (UID: \"e45b1572-6519-4c71-884a-115eeddd86cf\") " pod="e2e-models-e2e-worker-w4-50d738/distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r"
Nov 03 22:27:17.586395 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:17.586378    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e45b1572-6519-4c71-884a-115eeddd86cf-dshm\") pod \"distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r\" (UID: \"e45b1572-6519-4c71-884a-115eeddd86cf\") " pod="e2e-models-e2e-worker-w4-50d738/distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r"
Nov 03 22:27:17.586463 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:17.586446    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e45b1572-6519-4c71-884a-115eeddd86cf-tls-certs\") pod \"distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r\" (UID: \"e45b1572-6519-4c71-884a-115eeddd86cf\") " pod="e2e-models-e2e-worker-w4-50d738/distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r"
Nov 03 22:27:17.586646 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:17.586617    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e45b1572-6519-4c71-884a-115eeddd86cf-model-cache\") pod \"distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r\" (UID: \"e45b1572-6519-4c71-884a-115eeddd86cf\") " pod="e2e-models-e2e-worker-w4-50d738/distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r"
Nov 03 22:27:17.586702 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:17.586673    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e45b1572-6519-4c71-884a-115eeddd86cf-home\") pod \"distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r\" (UID: \"e45b1572-6519-4c71-884a-115eeddd86cf\") " pod="e2e-models-e2e-worker-w4-50d738/distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r"
Nov 03 22:27:17.687617 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:17.687522    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e45b1572-6519-4c71-884a-115eeddd86cf-model-cache\") pod \"distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r\" (UID: \"e45b1572-6519-4c71-884a-115eeddd86cf\") " pod="e2e-models-e2e-worker-w4-50d738/distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r"
Nov 03 22:27:17.687874 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:17.687852    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e45b1572-6519-4c71-884a-115eeddd86cf-home\") pod \"distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r\" (UID: \"e45b1572-6519-4c71-884a-115eeddd86cf\") " pod="e2e-models-e2e-worker-w4-50d738/distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r"
Nov 03 22:27:17.687982 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:17.687956    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e45b1572-6519-4c71-884a-115eeddd86cf-model-cache\") pod \"distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r\" (UID: \"e45b1572-6519-4c71-884a-115eeddd86cf\") " pod="e2e-models-e2e-worker-w4-50d738/distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r"
Nov 03 22:27:17.688129 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:17.688073    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e45b1572-6519-4c71-884a-115eeddd86cf-tmp-dir\") pod \"distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r\" (UID: \"e45b1572-6519-4c71-884a-115eeddd86cf\") " pod="e2e-models-e2e-worker-w4-50d738/distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r"
Nov 03 22:27:17.688129 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:17.688122    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-428n5\" (UniqueName: \"kubernetes.io/projected/e45b1572-6519-4c71-884a-115eeddd86cf-kube-api-access-428n5\") pod \"distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r\" (UID: \"e45b1572-6519-4c71-884a-115eeddd86cf\") " pod="e2e-models-e2e-worker-w4-50d738/distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r"
Nov 03 22:27:17.688258 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:17.688159    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e45b1572-6519-4c71-884a-115eeddd86cf-dshm\") pod \"distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r\" (UID: \"e45b1572-6519-4c71-884a-115eeddd86cf\") " pod="e2e-models-e2e-worker-w4-50d738/distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r"
Nov 03 22:27:17.688258 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:17.688200    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e45b1572-6519-4c71-884a-115eeddd86cf-tls-certs\") pod \"distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r\" (UID: \"e45b1572-6519-4c71-884a-115eeddd86cf\") " pod="e2e-models-e2e-worker-w4-50d738/distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r"
Nov 03 22:27:17.688374 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:17.688258    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e45b1572-6519-4c71-884a-115eeddd86cf-home\") pod \"distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r\" (UID: \"e45b1572-6519-4c71-884a-115eeddd86cf\") " pod="e2e-models-e2e-worker-w4-50d738/distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r"
Nov 03 22:27:17.688826 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:17.688805    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e45b1572-6519-4c71-884a-115eeddd86cf-tmp-dir\") pod \"distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r\" (UID: \"e45b1572-6519-4c71-884a-115eeddd86cf\") " pod="e2e-models-e2e-worker-w4-50d738/distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r"
Nov 03 22:27:17.691483 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:17.691460    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e45b1572-6519-4c71-884a-115eeddd86cf-dshm\") pod \"distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r\" (UID: \"e45b1572-6519-4c71-884a-115eeddd86cf\") " pod="e2e-models-e2e-worker-w4-50d738/distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r"
Nov 03 22:27:17.693931 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:17.693897    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e45b1572-6519-4c71-884a-115eeddd86cf-tls-certs\") pod \"distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r\" (UID: \"e45b1572-6519-4c71-884a-115eeddd86cf\") " pod="e2e-models-e2e-worker-w4-50d738/distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r"
Nov 03 22:27:17.711834 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:17.711796    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-428n5\" (UniqueName: \"kubernetes.io/projected/e45b1572-6519-4c71-884a-115eeddd86cf-kube-api-access-428n5\") pod \"distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r\" (UID: \"e45b1572-6519-4c71-884a-115eeddd86cf\") " pod="e2e-models-e2e-worker-w4-50d738/distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r"
Nov 03 22:27:17.739645 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:17.739592    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-50d738/distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r"
Nov 03 22:27:18.045042 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:18.045011    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-50d738/distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r"]
Nov 03 22:27:18.046209 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:27:18.046177    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode45b1572_6519_4c71_884a_115eeddd86cf.slice/crio-8ec25817f8d33db3aa205f78992c12b050887ea6d446c20a233010e2dd033b80 WatchSource:0}: Error finding container 8ec25817f8d33db3aa205f78992c12b050887ea6d446c20a233010e2dd033b80: Status 404 returned error can't find the container with id 8ec25817f8d33db3aa205f78992c12b050887ea6d446c20a233010e2dd033b80
Nov 03 22:27:18.267060 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:18.267027    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-50d738/distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r" event={"ID":"e45b1572-6519-4c71-884a-115eeddd86cf","Type":"ContainerStarted","Data":"1eb0d38257f07af06dd086ccda66f291c38edb863e381ec546a7b5f46b1bc28a"}
Nov 03 22:27:18.267226 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:18.267069    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-50d738/distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r" event={"ID":"e45b1572-6519-4c71-884a-115eeddd86cf","Type":"ContainerStarted","Data":"8ec25817f8d33db3aa205f78992c12b050887ea6d446c20a233010e2dd033b80"}
Nov 03 22:27:18.309792 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:18.309701    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-50d738/distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r" podStartSLOduration=1.30968728 podStartE2EDuration="1.30968728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:27:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:27:18.306377146 +0000 UTC m=+862.332233008" watchObservedRunningTime="2026-11-03 22:27:18.30968728 +0000 UTC m=+862.335543142"
Nov 03 22:27:18.423045 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:18.423006    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:27:18.739827 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:18.739801    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-50d738/distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r"
Nov 03 22:27:20.020802 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:20.020770    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:27:21.622972 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:21.622939    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:27:23.119993 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:23.119955    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:27:24.900762 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:24.900727    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:27:26.623363 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:26.623325    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:27:27.740286 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:27.740250    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-50d738/distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r"
Nov 03 22:27:27.782253 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:27.773554    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-50d738/distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r"
Nov 03 22:27:28.201092 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:28.201051    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9fdlh"]
Nov 03 22:27:28.206253 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:28.206207    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9fdlh"
Nov 03 22:27:28.246237 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:28.244757    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9fdlh"]
Nov 03 22:27:28.306008 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:28.305969    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a625b54-3dd0-4a42-9583-fe46b592c1a6-istio-token\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-9fdlh\" (UID: \"9a625b54-3dd0-4a42-9583-fe46b592c1a6\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9fdlh"
Nov 03 22:27:28.306202 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:28.306099    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngwjt\" (UniqueName: \"kubernetes.io/projected/9a625b54-3dd0-4a42-9583-fe46b592c1a6-kube-api-access-ngwjt\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-9fdlh\" (UID: \"9a625b54-3dd0-4a42-9583-fe46b592c1a6\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9fdlh"
Nov 03 22:27:28.306202 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:28.306146    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a625b54-3dd0-4a42-9583-fe46b592c1a6-workload-socket\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-9fdlh\" (UID: \"9a625b54-3dd0-4a42-9583-fe46b592c1a6\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9fdlh"
Nov 03 22:27:28.306202 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:28.306170    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a625b54-3dd0-4a42-9583-fe46b592c1a6-istio-data\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-9fdlh\" (UID: \"9a625b54-3dd0-4a42-9583-fe46b592c1a6\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9fdlh"
Nov 03 22:27:28.306387 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:28.306365    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a625b54-3dd0-4a42-9583-fe46b592c1a6-istio-envoy\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-9fdlh\" (UID: \"9a625b54-3dd0-4a42-9583-fe46b592c1a6\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9fdlh"
Nov 03 22:27:28.306455 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:28.306441    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a625b54-3dd0-4a42-9583-fe46b592c1a6-istio-podinfo\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-9fdlh\" (UID: \"9a625b54-3dd0-4a42-9583-fe46b592c1a6\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9fdlh"
Nov 03 22:27:28.306496 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:28.306470    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a625b54-3dd0-4a42-9583-fe46b592c1a6-workload-certs\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-9fdlh\" (UID: \"9a625b54-3dd0-4a42-9583-fe46b592c1a6\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9fdlh"
Nov 03 22:27:28.306551 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:28.306494    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a625b54-3dd0-4a42-9583-fe46b592c1a6-credential-socket\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-9fdlh\" (UID: \"9a625b54-3dd0-4a42-9583-fe46b592c1a6\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9fdlh"
Nov 03 22:27:28.306659 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:28.306643    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a625b54-3dd0-4a42-9583-fe46b592c1a6-istiod-ca-cert\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-9fdlh\" (UID: \"9a625b54-3dd0-4a42-9583-fe46b592c1a6\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9fdlh"
Nov 03 22:27:28.346661 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:28.346627    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-50d738/distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r"
Nov 03 22:27:28.347600 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:28.347572    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:27:28.409368 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:28.408913    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a625b54-3dd0-4a42-9583-fe46b592c1a6-workload-socket\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-9fdlh\" (UID: \"9a625b54-3dd0-4a42-9583-fe46b592c1a6\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9fdlh"
Nov 03 22:27:28.409368 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:28.408967    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a625b54-3dd0-4a42-9583-fe46b592c1a6-istio-data\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-9fdlh\" (UID: \"9a625b54-3dd0-4a42-9583-fe46b592c1a6\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9fdlh"
Nov 03 22:27:28.409368 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:28.409041    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a625b54-3dd0-4a42-9583-fe46b592c1a6-istio-envoy\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-9fdlh\" (UID: \"9a625b54-3dd0-4a42-9583-fe46b592c1a6\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9fdlh"
Nov 03 22:27:28.409368 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:28.409113    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a625b54-3dd0-4a42-9583-fe46b592c1a6-istio-podinfo\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-9fdlh\" (UID: \"9a625b54-3dd0-4a42-9583-fe46b592c1a6\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9fdlh"
Nov 03 22:27:28.409368 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:28.409151    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a625b54-3dd0-4a42-9583-fe46b592c1a6-workload-certs\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-9fdlh\" (UID: \"9a625b54-3dd0-4a42-9583-fe46b592c1a6\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9fdlh"
Nov 03 22:27:28.409368 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:28.409178    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a625b54-3dd0-4a42-9583-fe46b592c1a6-credential-socket\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-9fdlh\" (UID: \"9a625b54-3dd0-4a42-9583-fe46b592c1a6\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9fdlh"
Nov 03 22:27:28.409368 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:28.409296    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a625b54-3dd0-4a42-9583-fe46b592c1a6-istiod-ca-cert\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-9fdlh\" (UID: \"9a625b54-3dd0-4a42-9583-fe46b592c1a6\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9fdlh"
Nov 03 22:27:28.410234 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:28.410135    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a625b54-3dd0-4a42-9583-fe46b592c1a6-istio-token\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-9fdlh\" (UID: \"9a625b54-3dd0-4a42-9583-fe46b592c1a6\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9fdlh"
Nov 03 22:27:28.410471 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:28.410400    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngwjt\" (UniqueName: \"kubernetes.io/projected/9a625b54-3dd0-4a42-9583-fe46b592c1a6-kube-api-access-ngwjt\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-9fdlh\" (UID: \"9a625b54-3dd0-4a42-9583-fe46b592c1a6\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9fdlh"
Nov 03 22:27:28.411637 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:28.411581    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a625b54-3dd0-4a42-9583-fe46b592c1a6-workload-socket\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-9fdlh\" (UID: \"9a625b54-3dd0-4a42-9583-fe46b592c1a6\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9fdlh"
Nov 03 22:27:28.412205 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:28.412154    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a625b54-3dd0-4a42-9583-fe46b592c1a6-istio-data\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-9fdlh\" (UID: \"9a625b54-3dd0-4a42-9583-fe46b592c1a6\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9fdlh"
Nov 03 22:27:28.413595 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:28.413572    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a625b54-3dd0-4a42-9583-fe46b592c1a6-credential-socket\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-9fdlh\" (UID: \"9a625b54-3dd0-4a42-9583-fe46b592c1a6\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9fdlh"
Nov 03 22:27:28.415469 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:28.414667    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a625b54-3dd0-4a42-9583-fe46b592c1a6-workload-certs\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-9fdlh\" (UID: \"9a625b54-3dd0-4a42-9583-fe46b592c1a6\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9fdlh"
Nov 03 22:27:28.415620 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:28.415339    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a625b54-3dd0-4a42-9583-fe46b592c1a6-istiod-ca-cert\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-9fdlh\" (UID: \"9a625b54-3dd0-4a42-9583-fe46b592c1a6\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9fdlh"
Nov 03 22:27:28.424656 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:28.424632    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a625b54-3dd0-4a42-9583-fe46b592c1a6-istio-podinfo\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-9fdlh\" (UID: \"9a625b54-3dd0-4a42-9583-fe46b592c1a6\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9fdlh"
Nov 03 22:27:28.425652 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:28.425633    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a625b54-3dd0-4a42-9583-fe46b592c1a6-istio-envoy\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-9fdlh\" (UID: \"9a625b54-3dd0-4a42-9583-fe46b592c1a6\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9fdlh"
Nov 03 22:27:28.433658 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:28.433627    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngwjt\" (UniqueName: \"kubernetes.io/projected/9a625b54-3dd0-4a42-9583-fe46b592c1a6-kube-api-access-ngwjt\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-9fdlh\" (UID: \"9a625b54-3dd0-4a42-9583-fe46b592c1a6\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9fdlh"
Nov 03 22:27:28.436578 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:28.436542    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a625b54-3dd0-4a42-9583-fe46b592c1a6-istio-token\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-9fdlh\" (UID: \"9a625b54-3dd0-4a42-9583-fe46b592c1a6\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9fdlh"
Nov 03 22:27:28.524256 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:28.524145    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9fdlh"
Nov 03 22:27:28.889038 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:28.889004    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9fdlh"]
Nov 03 22:27:28.895164 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:28.895123    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:27:28.895295 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:28.895227    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:27:28.895295 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:28.895277    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:27:28.921250 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:28.921195    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf"]
Nov 03 22:27:28.927566 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:28.927541    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf"
Nov 03 22:27:28.955335 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:28.954461    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf"]
Nov 03 22:27:29.121368 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:29.121332    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/489884ba-7c24-48f7-8a7e-e5494d3232f7-istiod-ca-cert\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf\" (UID: \"489884ba-7c24-48f7-8a7e-e5494d3232f7\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf"
Nov 03 22:27:29.121671 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:29.121648    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c5wt\" (UniqueName: \"kubernetes.io/projected/489884ba-7c24-48f7-8a7e-e5494d3232f7-kube-api-access-6c5wt\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf\" (UID: \"489884ba-7c24-48f7-8a7e-e5494d3232f7\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf"
Nov 03 22:27:29.121943 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:29.121922    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/489884ba-7c24-48f7-8a7e-e5494d3232f7-workload-certs\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf\" (UID: \"489884ba-7c24-48f7-8a7e-e5494d3232f7\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf"
Nov 03 22:27:29.122092 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:29.122076    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/489884ba-7c24-48f7-8a7e-e5494d3232f7-istio-podinfo\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf\" (UID: \"489884ba-7c24-48f7-8a7e-e5494d3232f7\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf"
Nov 03 22:27:29.122256 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:29.122237    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/489884ba-7c24-48f7-8a7e-e5494d3232f7-istio-envoy\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf\" (UID: \"489884ba-7c24-48f7-8a7e-e5494d3232f7\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf"
Nov 03 22:27:29.122393 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:29.122377    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/489884ba-7c24-48f7-8a7e-e5494d3232f7-istio-token\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf\" (UID: \"489884ba-7c24-48f7-8a7e-e5494d3232f7\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf"
Nov 03 22:27:29.122572 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:29.122555    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/489884ba-7c24-48f7-8a7e-e5494d3232f7-istio-data\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf\" (UID: \"489884ba-7c24-48f7-8a7e-e5494d3232f7\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf"
Nov 03 22:27:29.122713 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:29.122697    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/489884ba-7c24-48f7-8a7e-e5494d3232f7-credential-socket\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf\" (UID: \"489884ba-7c24-48f7-8a7e-e5494d3232f7\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf"
Nov 03 22:27:29.122865 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:29.122847    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/489884ba-7c24-48f7-8a7e-e5494d3232f7-workload-socket\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf\" (UID: \"489884ba-7c24-48f7-8a7e-e5494d3232f7\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf"
Nov 03 22:27:29.224089 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:29.224036    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/489884ba-7c24-48f7-8a7e-e5494d3232f7-istiod-ca-cert\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf\" (UID: \"489884ba-7c24-48f7-8a7e-e5494d3232f7\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf"
Nov 03 22:27:29.224311 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:29.224141    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c5wt\" (UniqueName: \"kubernetes.io/projected/489884ba-7c24-48f7-8a7e-e5494d3232f7-kube-api-access-6c5wt\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf\" (UID: \"489884ba-7c24-48f7-8a7e-e5494d3232f7\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf"
Nov 03 22:27:29.224311 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:29.224229    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/489884ba-7c24-48f7-8a7e-e5494d3232f7-workload-certs\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf\" (UID: \"489884ba-7c24-48f7-8a7e-e5494d3232f7\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf"
Nov 03 22:27:29.224311 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:29.224270    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/489884ba-7c24-48f7-8a7e-e5494d3232f7-istio-podinfo\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf\" (UID: \"489884ba-7c24-48f7-8a7e-e5494d3232f7\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf"
Nov 03 22:27:29.224506 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:29.224320    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/489884ba-7c24-48f7-8a7e-e5494d3232f7-istio-envoy\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf\" (UID: \"489884ba-7c24-48f7-8a7e-e5494d3232f7\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf"
Nov 03 22:27:29.224506 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:29.224347    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/489884ba-7c24-48f7-8a7e-e5494d3232f7-istio-token\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf\" (UID: \"489884ba-7c24-48f7-8a7e-e5494d3232f7\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf"
Nov 03 22:27:29.224506 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:29.224428    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/489884ba-7c24-48f7-8a7e-e5494d3232f7-istio-data\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf\" (UID: \"489884ba-7c24-48f7-8a7e-e5494d3232f7\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf"
Nov 03 22:27:29.224506 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:29.224471    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/489884ba-7c24-48f7-8a7e-e5494d3232f7-credential-socket\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf\" (UID: \"489884ba-7c24-48f7-8a7e-e5494d3232f7\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf"
Nov 03 22:27:29.224506 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:29.224503    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/489884ba-7c24-48f7-8a7e-e5494d3232f7-workload-socket\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf\" (UID: \"489884ba-7c24-48f7-8a7e-e5494d3232f7\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf"
Nov 03 22:27:29.224924 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:29.224894    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/489884ba-7c24-48f7-8a7e-e5494d3232f7-istiod-ca-cert\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf\" (UID: \"489884ba-7c24-48f7-8a7e-e5494d3232f7\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf"
Nov 03 22:27:29.225771 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:29.225748    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/489884ba-7c24-48f7-8a7e-e5494d3232f7-workload-socket\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf\" (UID: \"489884ba-7c24-48f7-8a7e-e5494d3232f7\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf"
Nov 03 22:27:29.225968 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:29.225939    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/489884ba-7c24-48f7-8a7e-e5494d3232f7-credential-socket\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf\" (UID: \"489884ba-7c24-48f7-8a7e-e5494d3232f7\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf"
Nov 03 22:27:29.226108 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:29.226088    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/489884ba-7c24-48f7-8a7e-e5494d3232f7-istio-data\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf\" (UID: \"489884ba-7c24-48f7-8a7e-e5494d3232f7\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf"
Nov 03 22:27:29.226171 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:29.226092    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/489884ba-7c24-48f7-8a7e-e5494d3232f7-workload-certs\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf\" (UID: \"489884ba-7c24-48f7-8a7e-e5494d3232f7\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf"
Nov 03 22:27:29.230259 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:29.227944    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/489884ba-7c24-48f7-8a7e-e5494d3232f7-istio-envoy\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf\" (UID: \"489884ba-7c24-48f7-8a7e-e5494d3232f7\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf"
Nov 03 22:27:29.230259 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:29.229643    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/489884ba-7c24-48f7-8a7e-e5494d3232f7-istio-podinfo\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf\" (UID: \"489884ba-7c24-48f7-8a7e-e5494d3232f7\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf"
Nov 03 22:27:29.257075 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:29.256990    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c5wt\" (UniqueName: \"kubernetes.io/projected/489884ba-7c24-48f7-8a7e-e5494d3232f7-kube-api-access-6c5wt\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf\" (UID: \"489884ba-7c24-48f7-8a7e-e5494d3232f7\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf"
Nov 03 22:27:29.257751 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:29.257688    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/489884ba-7c24-48f7-8a7e-e5494d3232f7-istio-token\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf\" (UID: \"489884ba-7c24-48f7-8a7e-e5494d3232f7\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf"
Nov 03 22:27:29.314245 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:29.313530    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf"
Nov 03 22:27:29.337463 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:29.337426    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9fdlh" event={"ID":"9a625b54-3dd0-4a42-9583-fe46b592c1a6","Type":"ContainerStarted","Data":"fbcd896eba36a903dbcf4a1e2284b12c4a6807dd92b7b707f876eadafa858038"}
Nov 03 22:27:29.337636 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:29.337497    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9fdlh" event={"ID":"9a625b54-3dd0-4a42-9583-fe46b592c1a6","Type":"ContainerStarted","Data":"34f77a7a8b6ad645a198692dbf4195b5193eab14d7325eb4a0c250c018449570"}
Nov 03 22:27:29.537403 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:29.537236    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9fdlh"
Nov 03 22:27:29.819138 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:29.817591    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9fdlh" podStartSLOduration=1.8175700350000001 podStartE2EDuration="1.817570035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:27:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:27:29.419466414 +0000 UTC m=+873.445322278" watchObservedRunningTime="2026-11-03 22:27:29.817570035 +0000 UTC m=+873.843425897"
Nov 03 22:27:29.823041 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:29.823001    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:27:29.825062 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:29.825033    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:27:29.825283 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:29.825258    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:27:29.825817 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:29.823582    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf"]
Nov 03 22:27:30.353305 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:30.353270    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf" event={"ID":"489884ba-7c24-48f7-8a7e-e5494d3232f7","Type":"ContainerStarted","Data":"ff3b35ba33964b4d4718feb61bf79020cd3b07bb1453cd2898a3dc84892f8327"}
Nov 03 22:27:30.353961 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:30.353930    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf" event={"ID":"489884ba-7c24-48f7-8a7e-e5494d3232f7","Type":"ContainerStarted","Data":"86b4dc28811306b6ef13e8039612661250b580a7fd0b1c8a3c6dd03ee1eeab6b"}
Nov 03 22:27:30.408449 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:30.408387    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf" podStartSLOduration=2.40836636 podStartE2EDuration="2.40836636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:27:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:27:30.403959073 +0000 UTC m=+874.429814936" watchObservedRunningTime="2026-11-03 22:27:30.40836636 +0000 UTC m=+874.434222224"
Nov 03 22:27:30.435083 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:30.435030    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:27:30.526431 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:30.526394    4431 patch_prober.go:29] interesting pod/e2e-worker-w2-553753-openshift-default-558949869c-9fdlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 03 22:27:30.526603 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:30.526460    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9fdlh" podUID="9a625b54-3dd0-4a42-9583-fe46b592c1a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 03 22:27:30.577101 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:30.576977    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:27:31.305645 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:27:31.305601    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:27:31.314885 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:31.314850    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf"
Nov 03 22:27:31.316477 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:31.316449    4431 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Nov 03 22:27:31.316585 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:31.316508    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf" podUID="489884ba-7c24-48f7-8a7e-e5494d3232f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Nov 03 22:27:31.524898 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:31.524850    4431 patch_prober.go:29] interesting pod/e2e-worker-w2-553753-openshift-default-558949869c-9fdlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 03 22:27:31.525332 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:31.524915    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9fdlh" podUID="9a625b54-3dd0-4a42-9583-fe46b592c1a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 03 22:27:32.315050 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:32.315012    4431 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Nov 03 22:27:32.315205 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:32.315061    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf" podUID="489884ba-7c24-48f7-8a7e-e5494d3232f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Nov 03 22:27:32.415846 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:32.415805    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:27:32.525468 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:32.525439    4431 patch_prober.go:29] interesting pod/e2e-worker-w2-553753-openshift-default-558949869c-9fdlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 03 22:27:32.525919 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:32.525895    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9fdlh" podUID="9a625b54-3dd0-4a42-9583-fe46b592c1a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 03 22:27:33.315000 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:33.314967    4431 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Nov 03 22:27:33.315164 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:33.315018    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf" podUID="489884ba-7c24-48f7-8a7e-e5494d3232f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Nov 03 22:27:33.524996 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:33.524955    4431 patch_prober.go:29] interesting pod/e2e-worker-w2-553753-openshift-default-558949869c-9fdlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 03 22:27:33.525176 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:33.525018    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9fdlh" podUID="9a625b54-3dd0-4a42-9583-fe46b592c1a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 03 22:27:34.261242 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:34.259730    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:27:34.315511 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:34.315071    4431 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Nov 03 22:27:34.315511 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:34.315130    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf" podUID="489884ba-7c24-48f7-8a7e-e5494d3232f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Nov 03 22:27:34.525566 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:34.525471    4431 patch_prober.go:29] interesting pod/e2e-worker-w2-553753-openshift-default-558949869c-9fdlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 03 22:27:34.525566 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:34.525528    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9fdlh" podUID="9a625b54-3dd0-4a42-9583-fe46b592c1a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 03 22:27:35.314888 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:35.314840    4431 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Nov 03 22:27:35.315332 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:35.314912    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf" podUID="489884ba-7c24-48f7-8a7e-e5494d3232f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Nov 03 22:27:35.525038 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:35.524998    4431 patch_prober.go:29] interesting pod/e2e-worker-w2-553753-openshift-default-558949869c-9fdlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 03 22:27:35.525208 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:35.525064    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9fdlh" podUID="9a625b54-3dd0-4a42-9583-fe46b592c1a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 03 22:27:36.013938 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:36.013889    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:27:36.242558 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:36.242519    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:27:36.314726 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:36.314619    4431 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Nov 03 22:27:36.315046 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:36.315011    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf" podUID="489884ba-7c24-48f7-8a7e-e5494d3232f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Nov 03 22:27:36.888193 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:36.888165    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9fdlh"
Nov 03 22:27:36.888587 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:36.888566    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9fdlh"
Nov 03 22:27:36.889360 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:36.889339    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9fdlh"
Nov 03 22:27:37.612925 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:37.612895    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf"
Nov 03 22:27:37.613357 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:37.613340    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf"
Nov 03 22:27:37.614201 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:37.614182    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf"
Nov 03 22:27:37.913747 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:37.913713    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:27:39.544495 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:39.544458    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:27:41.023237 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:41.023125    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:27:42.635734 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:42.635690    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:27:43.161916 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:43.161888    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v"]
Nov 03 22:27:43.165972 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:43.165949    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v"
Nov 03 22:27:43.197481 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:43.197452    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v"]
Nov 03 22:27:43.232271 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:43.232236    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e928b67-e047-414b-92a2-8fc0792eed46-istio-data\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v\" (UID: \"1e928b67-e047-414b-92a2-8fc0792eed46\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v"
Nov 03 22:27:43.232450 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:43.232357    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e928b67-e047-414b-92a2-8fc0792eed46-istiod-ca-cert\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v\" (UID: \"1e928b67-e047-414b-92a2-8fc0792eed46\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v"
Nov 03 22:27:43.232450 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:43.232417    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e928b67-e047-414b-92a2-8fc0792eed46-workload-socket\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v\" (UID: \"1e928b67-e047-414b-92a2-8fc0792eed46\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v"
Nov 03 22:27:43.232586 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:43.232563    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e928b67-e047-414b-92a2-8fc0792eed46-credential-socket\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v\" (UID: \"1e928b67-e047-414b-92a2-8fc0792eed46\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v"
Nov 03 22:27:43.232630 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:43.232615    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e928b67-e047-414b-92a2-8fc0792eed46-istio-envoy\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v\" (UID: \"1e928b67-e047-414b-92a2-8fc0792eed46\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v"
Nov 03 22:27:43.232700 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:43.232682    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlmlf\" (UniqueName: \"kubernetes.io/projected/1e928b67-e047-414b-92a2-8fc0792eed46-kube-api-access-qlmlf\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v\" (UID: \"1e928b67-e047-414b-92a2-8fc0792eed46\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v"
Nov 03 22:27:43.232740 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:43.232715    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e928b67-e047-414b-92a2-8fc0792eed46-istio-podinfo\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v\" (UID: \"1e928b67-e047-414b-92a2-8fc0792eed46\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v"
Nov 03 22:27:43.232783 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:43.232742    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e928b67-e047-414b-92a2-8fc0792eed46-istio-token\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v\" (UID: \"1e928b67-e047-414b-92a2-8fc0792eed46\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v"
Nov 03 22:27:43.232910 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:43.232892    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e928b67-e047-414b-92a2-8fc0792eed46-workload-certs\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v\" (UID: \"1e928b67-e047-414b-92a2-8fc0792eed46\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v"
Nov 03 22:27:43.333733 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:43.333692    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e928b67-e047-414b-92a2-8fc0792eed46-workload-socket\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v\" (UID: \"1e928b67-e047-414b-92a2-8fc0792eed46\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v"
Nov 03 22:27:43.333983 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:43.333780    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e928b67-e047-414b-92a2-8fc0792eed46-credential-socket\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v\" (UID: \"1e928b67-e047-414b-92a2-8fc0792eed46\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v"
Nov 03 22:27:43.333983 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:43.333816    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e928b67-e047-414b-92a2-8fc0792eed46-istio-envoy\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v\" (UID: \"1e928b67-e047-414b-92a2-8fc0792eed46\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v"
Nov 03 22:27:43.333983 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:43.333844    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlmlf\" (UniqueName: \"kubernetes.io/projected/1e928b67-e047-414b-92a2-8fc0792eed46-kube-api-access-qlmlf\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v\" (UID: \"1e928b67-e047-414b-92a2-8fc0792eed46\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v"
Nov 03 22:27:43.333983 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:43.333876    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e928b67-e047-414b-92a2-8fc0792eed46-istio-podinfo\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v\" (UID: \"1e928b67-e047-414b-92a2-8fc0792eed46\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v"
Nov 03 22:27:43.333983 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:43.333913    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e928b67-e047-414b-92a2-8fc0792eed46-istio-token\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v\" (UID: \"1e928b67-e047-414b-92a2-8fc0792eed46\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v"
Nov 03 22:27:43.334343 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:43.334012    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e928b67-e047-414b-92a2-8fc0792eed46-workload-certs\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v\" (UID: \"1e928b67-e047-414b-92a2-8fc0792eed46\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v"
Nov 03 22:27:43.334343 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:43.334131    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e928b67-e047-414b-92a2-8fc0792eed46-istio-data\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v\" (UID: \"1e928b67-e047-414b-92a2-8fc0792eed46\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v"
Nov 03 22:27:43.334343 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:43.334197    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e928b67-e047-414b-92a2-8fc0792eed46-istiod-ca-cert\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v\" (UID: \"1e928b67-e047-414b-92a2-8fc0792eed46\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v"
Nov 03 22:27:43.334343 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:43.334208    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e928b67-e047-414b-92a2-8fc0792eed46-credential-socket\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v\" (UID: \"1e928b67-e047-414b-92a2-8fc0792eed46\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v"
Nov 03 22:27:43.334563 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:43.334507    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e928b67-e047-414b-92a2-8fc0792eed46-workload-socket\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v\" (UID: \"1e928b67-e047-414b-92a2-8fc0792eed46\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v"
Nov 03 22:27:43.334563 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:43.334548    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e928b67-e047-414b-92a2-8fc0792eed46-istio-data\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v\" (UID: \"1e928b67-e047-414b-92a2-8fc0792eed46\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v"
Nov 03 22:27:43.334851 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:43.334815    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e928b67-e047-414b-92a2-8fc0792eed46-workload-certs\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v\" (UID: \"1e928b67-e047-414b-92a2-8fc0792eed46\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v"
Nov 03 22:27:43.335031 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:43.335011    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e928b67-e047-414b-92a2-8fc0792eed46-istiod-ca-cert\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v\" (UID: \"1e928b67-e047-414b-92a2-8fc0792eed46\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v"
Nov 03 22:27:43.336583 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:43.336561    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e928b67-e047-414b-92a2-8fc0792eed46-istio-envoy\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v\" (UID: \"1e928b67-e047-414b-92a2-8fc0792eed46\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v"
Nov 03 22:27:43.336789 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:43.336768    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e928b67-e047-414b-92a2-8fc0792eed46-istio-podinfo\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v\" (UID: \"1e928b67-e047-414b-92a2-8fc0792eed46\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v"
Nov 03 22:27:43.357104 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:43.357074    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e928b67-e047-414b-92a2-8fc0792eed46-istio-token\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v\" (UID: \"1e928b67-e047-414b-92a2-8fc0792eed46\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v"
Nov 03 22:27:43.357352 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:43.357332    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlmlf\" (UniqueName: \"kubernetes.io/projected/1e928b67-e047-414b-92a2-8fc0792eed46-kube-api-access-qlmlf\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v\" (UID: \"1e928b67-e047-414b-92a2-8fc0792eed46\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v"
Nov 03 22:27:43.482626 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:43.482551    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v"
Nov 03 22:27:43.801987 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:43.801960    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v"]
Nov 03 22:27:43.803843 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:27:43.803821    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e928b67_e047_414b_92a2_8fc0792eed46.slice/crio-717447bedadc16d9cfb7cb3b2ef3344bfcb11a7e1839e3e476ec07c6155c273e WatchSource:0}: Error finding container 717447bedadc16d9cfb7cb3b2ef3344bfcb11a7e1839e3e476ec07c6155c273e: Status 404 returned error can't find the container with id 717447bedadc16d9cfb7cb3b2ef3344bfcb11a7e1839e3e476ec07c6155c273e
Nov 03 22:27:43.806474 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:43.806444    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:27:43.806554 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:43.806508    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:27:43.806554 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:43.806537    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:27:43.845559 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:43.845522    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv"]
Nov 03 22:27:43.851341 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:43.851317    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv"
Nov 03 22:27:43.877843 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:43.877816    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv"]
Nov 03 22:27:43.942556 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:43.942517    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27596c43-8204-4fd0-9ab0-612997367c40-istiod-ca-cert\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv\" (UID: \"27596c43-8204-4fd0-9ab0-612997367c40\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv"
Nov 03 22:27:43.942745 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:43.942580    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27596c43-8204-4fd0-9ab0-612997367c40-workload-socket\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv\" (UID: \"27596c43-8204-4fd0-9ab0-612997367c40\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv"
Nov 03 22:27:43.942745 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:43.942694    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27596c43-8204-4fd0-9ab0-612997367c40-istio-envoy\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv\" (UID: \"27596c43-8204-4fd0-9ab0-612997367c40\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv"
Nov 03 22:27:43.942864 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:43.942811    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27596c43-8204-4fd0-9ab0-612997367c40-istio-token\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv\" (UID: \"27596c43-8204-4fd0-9ab0-612997367c40\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv"
Nov 03 22:27:43.942923 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:43.942864    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27596c43-8204-4fd0-9ab0-612997367c40-istio-data\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv\" (UID: \"27596c43-8204-4fd0-9ab0-612997367c40\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv"
Nov 03 22:27:43.943000 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:43.942983    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69gkv\" (UniqueName: \"kubernetes.io/projected/27596c43-8204-4fd0-9ab0-612997367c40-kube-api-access-69gkv\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv\" (UID: \"27596c43-8204-4fd0-9ab0-612997367c40\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv"
Nov 03 22:27:43.943059 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:43.943032    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27596c43-8204-4fd0-9ab0-612997367c40-workload-certs\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv\" (UID: \"27596c43-8204-4fd0-9ab0-612997367c40\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv"
Nov 03 22:27:43.943205 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:43.943178    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27596c43-8204-4fd0-9ab0-612997367c40-istio-podinfo\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv\" (UID: \"27596c43-8204-4fd0-9ab0-612997367c40\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv"
Nov 03 22:27:43.943304 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:43.943281    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27596c43-8204-4fd0-9ab0-612997367c40-credential-socket\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv\" (UID: \"27596c43-8204-4fd0-9ab0-612997367c40\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv"
Nov 03 22:27:44.045089 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:44.045054    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69gkv\" (UniqueName: \"kubernetes.io/projected/27596c43-8204-4fd0-9ab0-612997367c40-kube-api-access-69gkv\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv\" (UID: \"27596c43-8204-4fd0-9ab0-612997367c40\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv"
Nov 03 22:27:44.045089 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:44.045105    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27596c43-8204-4fd0-9ab0-612997367c40-workload-certs\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv\" (UID: \"27596c43-8204-4fd0-9ab0-612997367c40\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv"
Nov 03 22:27:44.045377 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:44.045169    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27596c43-8204-4fd0-9ab0-612997367c40-istio-podinfo\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv\" (UID: \"27596c43-8204-4fd0-9ab0-612997367c40\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv"
Nov 03 22:27:44.045377 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:44.045207    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27596c43-8204-4fd0-9ab0-612997367c40-credential-socket\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv\" (UID: \"27596c43-8204-4fd0-9ab0-612997367c40\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv"
Nov 03 22:27:44.045377 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:44.045280    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27596c43-8204-4fd0-9ab0-612997367c40-istiod-ca-cert\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv\" (UID: \"27596c43-8204-4fd0-9ab0-612997367c40\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv"
Nov 03 22:27:44.045377 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:44.045318    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27596c43-8204-4fd0-9ab0-612997367c40-workload-socket\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv\" (UID: \"27596c43-8204-4fd0-9ab0-612997367c40\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv"
Nov 03 22:27:44.045377 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:44.045355    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27596c43-8204-4fd0-9ab0-612997367c40-istio-envoy\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv\" (UID: \"27596c43-8204-4fd0-9ab0-612997367c40\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv"
Nov 03 22:27:44.045653 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:44.045433    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27596c43-8204-4fd0-9ab0-612997367c40-istio-token\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv\" (UID: \"27596c43-8204-4fd0-9ab0-612997367c40\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv"
Nov 03 22:27:44.045653 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:44.045461    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27596c43-8204-4fd0-9ab0-612997367c40-istio-data\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv\" (UID: \"27596c43-8204-4fd0-9ab0-612997367c40\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv"
Nov 03 22:27:44.045757 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:44.045682    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27596c43-8204-4fd0-9ab0-612997367c40-workload-certs\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv\" (UID: \"27596c43-8204-4fd0-9ab0-612997367c40\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv"
Nov 03 22:27:44.045816 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:44.045767    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27596c43-8204-4fd0-9ab0-612997367c40-istio-data\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv\" (UID: \"27596c43-8204-4fd0-9ab0-612997367c40\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv"
Nov 03 22:27:44.045946 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:44.045923    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27596c43-8204-4fd0-9ab0-612997367c40-credential-socket\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv\" (UID: \"27596c43-8204-4fd0-9ab0-612997367c40\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv"
Nov 03 22:27:44.046163 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:44.046135    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27596c43-8204-4fd0-9ab0-612997367c40-istiod-ca-cert\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv\" (UID: \"27596c43-8204-4fd0-9ab0-612997367c40\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv"
Nov 03 22:27:44.046351 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:44.046191    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27596c43-8204-4fd0-9ab0-612997367c40-workload-socket\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv\" (UID: \"27596c43-8204-4fd0-9ab0-612997367c40\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv"
Nov 03 22:27:44.048067 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:44.048043    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27596c43-8204-4fd0-9ab0-612997367c40-istio-envoy\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv\" (UID: \"27596c43-8204-4fd0-9ab0-612997367c40\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv"
Nov 03 22:27:44.048481 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:44.048460    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27596c43-8204-4fd0-9ab0-612997367c40-istio-podinfo\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv\" (UID: \"27596c43-8204-4fd0-9ab0-612997367c40\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv"
Nov 03 22:27:44.068047 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:44.067975    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69gkv\" (UniqueName: \"kubernetes.io/projected/27596c43-8204-4fd0-9ab0-612997367c40-kube-api-access-69gkv\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv\" (UID: \"27596c43-8204-4fd0-9ab0-612997367c40\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv"
Nov 03 22:27:44.068158 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:44.068077    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27596c43-8204-4fd0-9ab0-612997367c40-istio-token\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv\" (UID: \"27596c43-8204-4fd0-9ab0-612997367c40\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv"
Nov 03 22:27:44.127613 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:44.127577    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:27:44.183913 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:44.183877    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv"
Nov 03 22:27:44.470288 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:44.470250    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v" event={"ID":"1e928b67-e047-414b-92a2-8fc0792eed46","Type":"ContainerStarted","Data":"96198a4dd8065a2d9031b23b150da2e49e6eff60785e6216f90929120506961a"}
Nov 03 22:27:44.470481 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:44.470316    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v" event={"ID":"1e928b67-e047-414b-92a2-8fc0792eed46","Type":"ContainerStarted","Data":"717447bedadc16d9cfb7cb3b2ef3344bfcb11a7e1839e3e476ec07c6155c273e"}
Nov 03 22:27:44.483708 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:44.483678    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v"
Nov 03 22:27:44.485621 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:44.485598    4431 patch_prober.go:29] interesting pod/e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 03 22:27:44.485736 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:44.485642    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v" podUID="1e928b67-e047-414b-92a2-8fc0792eed46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 03 22:27:44.514280 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:44.514252    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv"]
Nov 03 22:27:44.516147 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:27:44.516120    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27596c43_8204_4fd0_9ab0_612997367c40.slice/crio-108d8c05dab33f5bfc5d0b0539fca223bdc6a007194c16baa01e1c4f12c593eb WatchSource:0}: Error finding container 108d8c05dab33f5bfc5d0b0539fca223bdc6a007194c16baa01e1c4f12c593eb: Status 404 returned error can't find the container with id 108d8c05dab33f5bfc5d0b0539fca223bdc6a007194c16baa01e1c4f12c593eb
Nov 03 22:27:44.518683 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:44.518653    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:27:44.518767 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:44.518719    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:27:44.518767 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:44.518755    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:27:45.483905 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:45.483871    4431 patch_prober.go:29] interesting pod/e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 03 22:27:45.484588 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:45.484555    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v" podUID="1e928b67-e047-414b-92a2-8fc0792eed46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 03 22:27:45.497249 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:45.497195    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv" event={"ID":"27596c43-8204-4fd0-9ab0-612997367c40","Type":"ContainerStarted","Data":"7724ccc321cb6f5134ba8959602ea50740164befaddf398b8ead2a4ae94de574"}
Nov 03 22:27:45.497439 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:45.497278    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv" event={"ID":"27596c43-8204-4fd0-9ab0-612997367c40","Type":"ContainerStarted","Data":"108d8c05dab33f5bfc5d0b0539fca223bdc6a007194c16baa01e1c4f12c593eb"}
Nov 03 22:27:45.548386 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:45.547525    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v" podStartSLOduration=2.547503501 podStartE2EDuration="2.547503501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:27:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:27:44.526074812 +0000 UTC m=+888.551930675" watchObservedRunningTime="2026-11-03 22:27:45.547503501 +0000 UTC m=+889.573359365"
Nov 03 22:27:45.740289 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:45.738678    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:27:45.992822 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:45.992718    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:27:46.193239 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:46.190914    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv"
Nov 03 22:27:46.193239 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:46.191038    4431 patch_prober.go:29] interesting pod/e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 03 22:27:46.193239 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:46.191083    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv" podUID="27596c43-8204-4fd0-9ab0-612997367c40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 03 22:27:46.485239 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:46.483684    4431 patch_prober.go:29] interesting pod/e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 03 22:27:46.485239 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:46.483748    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v" podUID="1e928b67-e047-414b-92a2-8fc0792eed46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 03 22:27:47.190815 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:47.185654    4431 patch_prober.go:29] interesting pod/e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 03 22:27:47.190815 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:47.185718    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv" podUID="27596c43-8204-4fd0-9ab0-612997367c40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 03 22:27:47.484297 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:47.483652    4431 patch_prober.go:29] interesting pod/e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 03 22:27:47.484297 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:47.483717    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v" podUID="1e928b67-e047-414b-92a2-8fc0792eed46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 03 22:27:47.718550 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:47.718513    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:27:48.189624 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:48.189586    4431 patch_prober.go:29] interesting pod/e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 03 22:27:48.189909 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:48.189878    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv" podUID="27596c43-8204-4fd0-9ab0-612997367c40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 03 22:27:48.483725 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:48.483628    4431 patch_prober.go:29] interesting pod/e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 03 22:27:48.484000 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:48.483963    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v" podUID="1e928b67-e047-414b-92a2-8fc0792eed46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 03 22:27:49.187265 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:49.186633    4431 patch_prober.go:29] interesting pod/e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 03 22:27:49.187265 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:49.186701    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv" podUID="27596c43-8204-4fd0-9ab0-612997367c40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 03 22:27:49.435891 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:49.435858    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:27:49.491727 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:49.491637    4431 patch_prober.go:29] interesting pod/e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 03 22:27:49.491973 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:49.491943    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v" podUID="1e928b67-e047-414b-92a2-8fc0792eed46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 03 22:27:50.184856 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:50.184821    4431 patch_prober.go:29] interesting pod/e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 03 22:27:50.185012 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:50.184877    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv" podUID="27596c43-8204-4fd0-9ab0-612997367c40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 03 22:27:50.483861 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:50.483768    4431 patch_prober.go:29] interesting pod/e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 03 22:27:50.483861 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:50.483831    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v" podUID="1e928b67-e047-414b-92a2-8fc0792eed46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 03 22:27:50.490426 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:50.490395    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:27:51.041423 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:51.041381    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:27:51.184801 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:51.184762    4431 patch_prober.go:29] interesting pod/e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 03 22:27:51.185006 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:51.184830    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv" podUID="27596c43-8204-4fd0-9ab0-612997367c40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 03 22:27:51.488058 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:51.488023    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v"
Nov 03 22:27:51.488503 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:51.488337    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v"
Nov 03 22:27:51.489191 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:51.489177    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v"
Nov 03 22:27:51.550736 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:51.550663    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv" podStartSLOduration=8.550644484 podStartE2EDuration="8.550644484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:27:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:27:45.554589658 +0000 UTC m=+889.580445520" watchObservedRunningTime="2026-11-03 22:27:51.550644484 +0000 UTC m=+895.576500410"
Nov 03 22:27:52.189035 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:52.188997    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv"
Nov 03 22:27:52.189310 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:52.189285    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv"
Nov 03 22:27:52.189925 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:52.189905    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv"
Nov 03 22:27:52.726608 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:52.726575    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:27:54.222333 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:54.222286    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:27:55.808036 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:55.807994    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:27:56.194466 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:56.194440    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mx5q8_ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea/ovn-acl-logging/0.log"
Nov 03 22:27:56.196884 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:56.196858    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mx5q8_ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea/ovn-acl-logging/0.log"
Nov 03 22:27:57.332181 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:57.332144    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:27:58.415812 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:58.415772    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x"]
Nov 03 22:27:58.432612 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:58.432578    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x"
Nov 03 22:27:58.446915 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:58.446889    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x"]
Nov 03 22:27:58.460294 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:58.460250    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-2d619f-openshift-default-dockercfg-2f4cd\""
Nov 03 22:27:58.533229 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:58.533194    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73504251-0779-459e-abfc-3d099b7a1339-credential-socket\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x\" (UID: \"73504251-0779-459e-abfc-3d099b7a1339\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x"
Nov 03 22:27:58.533410 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:58.533341    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73504251-0779-459e-abfc-3d099b7a1339-workload-certs\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x\" (UID: \"73504251-0779-459e-abfc-3d099b7a1339\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x"
Nov 03 22:27:58.533479 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:58.533463    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73504251-0779-459e-abfc-3d099b7a1339-istio-podinfo\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x\" (UID: \"73504251-0779-459e-abfc-3d099b7a1339\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x"
Nov 03 22:27:58.533542 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:58.533503    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p95m\" (UniqueName: \"kubernetes.io/projected/73504251-0779-459e-abfc-3d099b7a1339-kube-api-access-8p95m\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x\" (UID: \"73504251-0779-459e-abfc-3d099b7a1339\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x"
Nov 03 22:27:58.533649 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:58.533625    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73504251-0779-459e-abfc-3d099b7a1339-workload-socket\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x\" (UID: \"73504251-0779-459e-abfc-3d099b7a1339\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x"
Nov 03 22:27:58.533696 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:58.533673    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73504251-0779-459e-abfc-3d099b7a1339-istio-data\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x\" (UID: \"73504251-0779-459e-abfc-3d099b7a1339\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x"
Nov 03 22:27:58.533777 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:58.533762    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73504251-0779-459e-abfc-3d099b7a1339-istiod-ca-cert\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x\" (UID: \"73504251-0779-459e-abfc-3d099b7a1339\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x"
Nov 03 22:27:58.533856 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:58.533840    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73504251-0779-459e-abfc-3d099b7a1339-istio-envoy\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x\" (UID: \"73504251-0779-459e-abfc-3d099b7a1339\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x"
Nov 03 22:27:58.533912 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:58.533895    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73504251-0779-459e-abfc-3d099b7a1339-istio-token\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x\" (UID: \"73504251-0779-459e-abfc-3d099b7a1339\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x"
Nov 03 22:27:58.652692 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:58.649117    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73504251-0779-459e-abfc-3d099b7a1339-istiod-ca-cert\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x\" (UID: \"73504251-0779-459e-abfc-3d099b7a1339\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x"
Nov 03 22:27:58.652692 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:58.649267    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73504251-0779-459e-abfc-3d099b7a1339-istio-envoy\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x\" (UID: \"73504251-0779-459e-abfc-3d099b7a1339\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x"
Nov 03 22:27:58.652692 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:58.649321    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73504251-0779-459e-abfc-3d099b7a1339-istio-token\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x\" (UID: \"73504251-0779-459e-abfc-3d099b7a1339\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x"
Nov 03 22:27:58.652692 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:58.649370    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73504251-0779-459e-abfc-3d099b7a1339-credential-socket\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x\" (UID: \"73504251-0779-459e-abfc-3d099b7a1339\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x"
Nov 03 22:27:58.652692 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:58.649450    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73504251-0779-459e-abfc-3d099b7a1339-workload-certs\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x\" (UID: \"73504251-0779-459e-abfc-3d099b7a1339\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x"
Nov 03 22:27:58.652692 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:58.649534    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73504251-0779-459e-abfc-3d099b7a1339-istio-podinfo\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x\" (UID: \"73504251-0779-459e-abfc-3d099b7a1339\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x"
Nov 03 22:27:58.652692 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:58.649574    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p95m\" (UniqueName: \"kubernetes.io/projected/73504251-0779-459e-abfc-3d099b7a1339-kube-api-access-8p95m\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x\" (UID: \"73504251-0779-459e-abfc-3d099b7a1339\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x"
Nov 03 22:27:58.652692 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:58.649655    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73504251-0779-459e-abfc-3d099b7a1339-workload-socket\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x\" (UID: \"73504251-0779-459e-abfc-3d099b7a1339\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x"
Nov 03 22:27:58.652692 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:58.649693    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73504251-0779-459e-abfc-3d099b7a1339-istio-data\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x\" (UID: \"73504251-0779-459e-abfc-3d099b7a1339\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x"
Nov 03 22:27:58.652692 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:58.650303    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73504251-0779-459e-abfc-3d099b7a1339-istio-data\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x\" (UID: \"73504251-0779-459e-abfc-3d099b7a1339\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x"
Nov 03 22:27:58.654055 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:58.653704    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73504251-0779-459e-abfc-3d099b7a1339-istiod-ca-cert\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x\" (UID: \"73504251-0779-459e-abfc-3d099b7a1339\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x"
Nov 03 22:27:58.655099 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:58.654410    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73504251-0779-459e-abfc-3d099b7a1339-workload-certs\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x\" (UID: \"73504251-0779-459e-abfc-3d099b7a1339\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x"
Nov 03 22:27:58.655099 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:58.654690    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73504251-0779-459e-abfc-3d099b7a1339-credential-socket\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x\" (UID: \"73504251-0779-459e-abfc-3d099b7a1339\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x"
Nov 03 22:27:58.655099 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:58.654962    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73504251-0779-459e-abfc-3d099b7a1339-workload-socket\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x\" (UID: \"73504251-0779-459e-abfc-3d099b7a1339\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x"
Nov 03 22:27:58.659100 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:58.657402    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73504251-0779-459e-abfc-3d099b7a1339-istio-envoy\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x\" (UID: \"73504251-0779-459e-abfc-3d099b7a1339\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x"
Nov 03 22:27:58.660357 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:58.659820    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73504251-0779-459e-abfc-3d099b7a1339-istio-podinfo\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x\" (UID: \"73504251-0779-459e-abfc-3d099b7a1339\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x"
Nov 03 22:27:58.691437 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:58.691308    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p95m\" (UniqueName: \"kubernetes.io/projected/73504251-0779-459e-abfc-3d099b7a1339-kube-api-access-8p95m\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x\" (UID: \"73504251-0779-459e-abfc-3d099b7a1339\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x"
Nov 03 22:27:58.714244 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:58.705982    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73504251-0779-459e-abfc-3d099b7a1339-istio-token\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x\" (UID: \"73504251-0779-459e-abfc-3d099b7a1339\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x"
Nov 03 22:27:58.753813 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:58.753774    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x"
Nov 03 22:27:59.889247 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:59.878466    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x"]
Nov 03 22:27:59.917388 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:59.917344    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/simulator-w0-88a4d9-kserve-785f84888f-sjbzp" podUID="18bd460b-a0f1-415c-bcb4-41c9f70d2926" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:27:59.923828 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:27:59.923797    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73504251_0779_459e_abfc_3d099b7a1339.slice/crio-dca6fff42a51abfd2c1a824cc935a29a5b191827346c33c560090b6364fce937 WatchSource:0}: Error finding container dca6fff42a51abfd2c1a824cc935a29a5b191827346c33c560090b6364fce937: Status 404 returned error can't find the container with id dca6fff42a51abfd2c1a824cc935a29a5b191827346c33c560090b6364fce937
Nov 03 22:27:59.929245 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:59.929184    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:27:59.933292 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:59.933255    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:27:59.933548 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:59.933520    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:27:59.943249 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:27:59.938314    4431 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-88a4d9/simulator-w0-88a4d9-kserve-785f84888f-sjbzp" podUID="18bd460b-a0f1-415c-bcb4-41c9f70d2926" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:28:00.314493 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:00.310772    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:28:00.335237 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:00.334007    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-553753/distinct-2-w2-553753-kserve-5bf8568844-7q4l2" podUID="a1894b80-164c-4ce1-94f3-1e5011d9759e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:28:00.675050 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:00.675008    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x" event={"ID":"73504251-0779-459e-abfc-3d099b7a1339","Type":"ContainerStarted","Data":"bc05b9fb5baab8bfd239be42d22c2cc43804bc1cee4aa1a735eecc33c7843551"}
Nov 03 22:28:00.676583 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:00.676552    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x" event={"ID":"73504251-0779-459e-abfc-3d099b7a1339","Type":"ContainerStarted","Data":"dca6fff42a51abfd2c1a824cc935a29a5b191827346c33c560090b6364fce937"}
Nov 03 22:28:00.754498 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:00.754432    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x" podStartSLOduration=2.754411766 podStartE2EDuration="2.754411766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:27:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:28:00.748897187 +0000 UTC m=+904.774753053" watchObservedRunningTime="2026-11-03 22:28:00.754411766 +0000 UTC m=+904.780267629"
Nov 03 22:28:00.755133 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:00.754781    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x"
Nov 03 22:28:00.906242 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:00.905767    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-w6-2d619f-kserve-74dbc56695-tnvtb" podUID="90310d72-c818-4fdd-b4e7-b51f633caa05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:28:00.915605 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:00.915564    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-553753/simulator-w2-553753-kserve-64d77d4dfc-4rlbn" podUID="a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:28:00.916198 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:00.915831    4431 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-553753/simulator-w2-553753-kserve-64d77d4dfc-4rlbn" podUID="a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:28:01.014015 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:01.013916    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5187e1/premium-w3-5187e1-kserve-56bf8788d4-pp8st" podUID="a70ecfe9-020a-4331-b67c-aaaa46ed1eb1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:28:01.358293 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:28:01.356046    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:28:01.766240 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:01.765629    4431 patch_prober.go:29] interesting pod/e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Nov 03 22:28:01.766240 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:01.765690    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x" podUID="73504251-0779-459e-abfc-3d099b7a1339" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Nov 03 22:28:02.601209 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:02.601168    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:28:02.760847 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:02.760799    4431 patch_prober.go:29] interesting pod/e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Nov 03 22:28:02.761065 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:02.760886    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x" podUID="73504251-0779-459e-abfc-3d099b7a1339" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Nov 03 22:28:03.764353 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:03.764315    4431 patch_prober.go:29] interesting pod/e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Nov 03 22:28:03.764981 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:03.764948    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x" podUID="73504251-0779-459e-abfc-3d099b7a1339" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Nov 03 22:28:04.198894 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:04.198860    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:28:04.762651 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:04.762612    4431 patch_prober.go:29] interesting pod/e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Nov 03 22:28:04.762958 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:04.762925    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x" podUID="73504251-0779-459e-abfc-3d099b7a1339" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Nov 03 22:28:05.759082 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:05.759046    4431 patch_prober.go:29] interesting pod/e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Nov 03 22:28:05.759701 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:05.759652    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x" podUID="73504251-0779-459e-abfc-3d099b7a1339" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Nov 03 22:28:05.801156 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:05.801129    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:28:06.763414 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:06.763377    4431 patch_prober.go:29] interesting pod/e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Nov 03 22:28:06.764016 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:06.763982    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x" podUID="73504251-0779-459e-abfc-3d099b7a1339" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Nov 03 22:28:07.310634 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:07.310600    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:28:08.184262 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:08.183066    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x"
Nov 03 22:28:08.184262 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:08.183870    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x"
Nov 03 22:28:08.184803 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:08.184473    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x"
Nov 03 22:28:09.008479 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:09.008436    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:28:10.618048 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:10.618002    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:28:12.181605 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:12.181562    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:28:13.335896 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:13.335860    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-85x5q"]
Nov 03 22:28:13.351174 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:13.351143    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-85x5q"
Nov 03 22:28:13.361617 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:13.361592    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-85x5q"]
Nov 03 22:28:13.412036 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:13.412007    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dffba47-88ed-4357-862e-d5c6e1d34060-istio-data\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-85x5q\" (UID: \"6dffba47-88ed-4357-862e-d5c6e1d34060\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-85x5q"
Nov 03 22:28:13.412205 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:13.412134    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dffba47-88ed-4357-862e-d5c6e1d34060-workload-certs\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-85x5q\" (UID: \"6dffba47-88ed-4357-862e-d5c6e1d34060\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-85x5q"
Nov 03 22:28:13.412279 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:13.412242    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dffba47-88ed-4357-862e-d5c6e1d34060-workload-socket\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-85x5q\" (UID: \"6dffba47-88ed-4357-862e-d5c6e1d34060\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-85x5q"
Nov 03 22:28:13.412324 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:13.412276    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dffba47-88ed-4357-862e-d5c6e1d34060-credential-socket\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-85x5q\" (UID: \"6dffba47-88ed-4357-862e-d5c6e1d34060\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-85x5q"
Nov 03 22:28:13.412364 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:13.412327    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dffba47-88ed-4357-862e-d5c6e1d34060-istio-token\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-85x5q\" (UID: \"6dffba47-88ed-4357-862e-d5c6e1d34060\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-85x5q"
Nov 03 22:28:13.412364 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:13.412359    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dffba47-88ed-4357-862e-d5c6e1d34060-istiod-ca-cert\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-85x5q\" (UID: \"6dffba47-88ed-4357-862e-d5c6e1d34060\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-85x5q"
Nov 03 22:28:13.412427 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:13.412391    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dffba47-88ed-4357-862e-d5c6e1d34060-istio-envoy\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-85x5q\" (UID: \"6dffba47-88ed-4357-862e-d5c6e1d34060\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-85x5q"
Nov 03 22:28:13.412527 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:13.412511    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qm4cj\" (UniqueName: \"kubernetes.io/projected/6dffba47-88ed-4357-862e-d5c6e1d34060-kube-api-access-qm4cj\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-85x5q\" (UID: \"6dffba47-88ed-4357-862e-d5c6e1d34060\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-85x5q"
Nov 03 22:28:13.412565 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:13.412536    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dffba47-88ed-4357-862e-d5c6e1d34060-istio-podinfo\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-85x5q\" (UID: \"6dffba47-88ed-4357-862e-d5c6e1d34060\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-85x5q"
Nov 03 22:28:13.513862 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:13.513825    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm4cj\" (UniqueName: \"kubernetes.io/projected/6dffba47-88ed-4357-862e-d5c6e1d34060-kube-api-access-qm4cj\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-85x5q\" (UID: \"6dffba47-88ed-4357-862e-d5c6e1d34060\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-85x5q"
Nov 03 22:28:13.513862 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:13.513866    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dffba47-88ed-4357-862e-d5c6e1d34060-istio-podinfo\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-85x5q\" (UID: \"6dffba47-88ed-4357-862e-d5c6e1d34060\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-85x5q"
Nov 03 22:28:13.514076 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:13.513911    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dffba47-88ed-4357-862e-d5c6e1d34060-istio-data\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-85x5q\" (UID: \"6dffba47-88ed-4357-862e-d5c6e1d34060\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-85x5q"
Nov 03 22:28:13.514076 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:13.514023    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dffba47-88ed-4357-862e-d5c6e1d34060-workload-certs\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-85x5q\" (UID: \"6dffba47-88ed-4357-862e-d5c6e1d34060\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-85x5q"
Nov 03 22:28:13.514076 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:13.514072    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dffba47-88ed-4357-862e-d5c6e1d34060-workload-socket\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-85x5q\" (UID: \"6dffba47-88ed-4357-862e-d5c6e1d34060\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-85x5q"
Nov 03 22:28:13.514210 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:13.514099    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dffba47-88ed-4357-862e-d5c6e1d34060-credential-socket\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-85x5q\" (UID: \"6dffba47-88ed-4357-862e-d5c6e1d34060\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-85x5q"
Nov 03 22:28:13.514210 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:13.514123    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dffba47-88ed-4357-862e-d5c6e1d34060-istio-token\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-85x5q\" (UID: \"6dffba47-88ed-4357-862e-d5c6e1d34060\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-85x5q"
Nov 03 22:28:13.514210 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:13.514151    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dffba47-88ed-4357-862e-d5c6e1d34060-istiod-ca-cert\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-85x5q\" (UID: \"6dffba47-88ed-4357-862e-d5c6e1d34060\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-85x5q"
Nov 03 22:28:13.514210 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:13.514178    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dffba47-88ed-4357-862e-d5c6e1d34060-istio-envoy\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-85x5q\" (UID: \"6dffba47-88ed-4357-862e-d5c6e1d34060\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-85x5q"
Nov 03 22:28:13.514534 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:13.514510    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dffba47-88ed-4357-862e-d5c6e1d34060-istio-data\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-85x5q\" (UID: \"6dffba47-88ed-4357-862e-d5c6e1d34060\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-85x5q"
Nov 03 22:28:13.514621 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:13.514597    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dffba47-88ed-4357-862e-d5c6e1d34060-credential-socket\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-85x5q\" (UID: \"6dffba47-88ed-4357-862e-d5c6e1d34060\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-85x5q"
Nov 03 22:28:13.514674 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:13.514633    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dffba47-88ed-4357-862e-d5c6e1d34060-workload-certs\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-85x5q\" (UID: \"6dffba47-88ed-4357-862e-d5c6e1d34060\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-85x5q"
Nov 03 22:28:13.514674 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:13.514666    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dffba47-88ed-4357-862e-d5c6e1d34060-workload-socket\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-85x5q\" (UID: \"6dffba47-88ed-4357-862e-d5c6e1d34060\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-85x5q"
Nov 03 22:28:13.515056 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:13.515034    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dffba47-88ed-4357-862e-d5c6e1d34060-istiod-ca-cert\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-85x5q\" (UID: \"6dffba47-88ed-4357-862e-d5c6e1d34060\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-85x5q"
Nov 03 22:28:13.516537 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:13.516517    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dffba47-88ed-4357-862e-d5c6e1d34060-istio-envoy\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-85x5q\" (UID: \"6dffba47-88ed-4357-862e-d5c6e1d34060\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-85x5q"
Nov 03 22:28:13.516638 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:13.516580    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dffba47-88ed-4357-862e-d5c6e1d34060-istio-podinfo\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-85x5q\" (UID: \"6dffba47-88ed-4357-862e-d5c6e1d34060\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-85x5q"
Nov 03 22:28:13.552971 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:13.552942    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm4cj\" (UniqueName: \"kubernetes.io/projected/6dffba47-88ed-4357-862e-d5c6e1d34060-kube-api-access-qm4cj\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-85x5q\" (UID: \"6dffba47-88ed-4357-862e-d5c6e1d34060\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-85x5q"
Nov 03 22:28:13.556274 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:13.556251    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dffba47-88ed-4357-862e-d5c6e1d34060-istio-token\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-85x5q\" (UID: \"6dffba47-88ed-4357-862e-d5c6e1d34060\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-85x5q"
Nov 03 22:28:13.668132 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:13.668097    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-85x5q"
Nov 03 22:28:13.856276 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:13.856202    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:28:13.998948 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:13.998834    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-85x5q"]
Nov 03 22:28:14.001624 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:28:14.001595    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dffba47_88ed_4357_862e_d5c6e1d34060.slice/crio-8a03c11507dc25a6b310cc9686e9a181193b1b97b4e49c8d1a50e3252b6133bb WatchSource:0}: Error finding container 8a03c11507dc25a6b310cc9686e9a181193b1b97b4e49c8d1a50e3252b6133bb: Status 404 returned error can't find the container with id 8a03c11507dc25a6b310cc9686e9a181193b1b97b4e49c8d1a50e3252b6133bb
Nov 03 22:28:14.004776 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:14.004742    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:28:14.004891 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:14.004823    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:28:14.004891 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:14.004869    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:28:14.775486 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:14.775448    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-85x5q" event={"ID":"6dffba47-88ed-4357-862e-d5c6e1d34060","Type":"ContainerStarted","Data":"68a696bb0f8bec8e92449bc8c6ac53a1e84dcbd6fb8657cf4009a1df8f49428b"}
Nov 03 22:28:14.775935 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:14.775502    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-85x5q" event={"ID":"6dffba47-88ed-4357-862e-d5c6e1d34060","Type":"ContainerStarted","Data":"8a03c11507dc25a6b310cc9686e9a181193b1b97b4e49c8d1a50e3252b6133bb"}
Nov 03 22:28:14.822256 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:14.822193    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-85x5q" podStartSLOduration=1.822177531 podStartE2EDuration="1.822177531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:28:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:28:14.81909646 +0000 UTC m=+918.844952323" watchObservedRunningTime="2026-11-03 22:28:14.822177531 +0000 UTC m=+918.848033392"
Nov 03 22:28:15.559240 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:15.555943    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:28:15.669172 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:15.669138    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-85x5q"
Nov 03 22:28:15.671036 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:15.671003    4431 patch_prober.go:29] interesting pod/e2e-worker-w2-553753-openshift-default-558949869c-85x5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 22:28:15.671208 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:15.671063    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-85x5q" podUID="6dffba47-88ed-4357-862e-d5c6e1d34060" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 22:28:16.668533 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:16.668491    4431 patch_prober.go:29] interesting pod/e2e-worker-w2-553753-openshift-default-558949869c-85x5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 22:28:16.669031 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:16.668540    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-85x5q" podUID="6dffba47-88ed-4357-862e-d5c6e1d34060" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 22:28:17.237685 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:17.237646    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:28:17.669095 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:17.669060    4431 patch_prober.go:29] interesting pod/e2e-worker-w2-553753-openshift-default-558949869c-85x5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 22:28:17.669520 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:17.669121    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-85x5q" podUID="6dffba47-88ed-4357-862e-d5c6e1d34060" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 22:28:18.669204 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:18.669166    4431 patch_prober.go:29] interesting pod/e2e-worker-w2-553753-openshift-default-558949869c-85x5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 22:28:18.669694 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:18.669262    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-85x5q" podUID="6dffba47-88ed-4357-862e-d5c6e1d34060" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 22:28:19.024781 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:19.024687    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:28:19.668825 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:19.668787    4431 patch_prober.go:29] interesting pod/e2e-worker-w2-553753-openshift-default-558949869c-85x5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 22:28:19.669002 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:19.668864    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-85x5q" podUID="6dffba47-88ed-4357-862e-d5c6e1d34060" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 22:28:20.417193 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:20.417159    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:28:20.555109 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:20.555069    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:28:20.669061 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:20.668984    4431 patch_prober.go:29] interesting pod/e2e-worker-w2-553753-openshift-default-558949869c-85x5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 22:28:20.669061 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:20.669045    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-85x5q" podUID="6dffba47-88ed-4357-862e-d5c6e1d34060" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 22:28:20.807559 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:20.807516    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:28:21.672989 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:21.672957    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-85x5q"
Nov 03 22:28:21.673418 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:21.673403    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-85x5q"
Nov 03 22:28:21.674042 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:21.674023    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-85x5q"
Nov 03 22:28:22.683957 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:22.683918    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:28:24.324372 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:24.324325    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:28:25.943766 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:25.943727    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:28:27.535471 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:27.535434    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:28:28.685856 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:28.685815    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g"]
Nov 03 22:28:28.691948 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:28.691917    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g"
Nov 03 22:28:28.710233 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:28.710191    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g"]
Nov 03 22:28:28.725287 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:28.721520    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-434ce3-openshift-default-dockercfg-4lnm5\""
Nov 03 22:28:28.741302 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:28.740769    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cd86bdd-8465-4676-9491-2266a2e86b3a-istio-envoy\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g\" (UID: \"0cd86bdd-8465-4676-9491-2266a2e86b3a\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g"
Nov 03 22:28:28.741302 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:28.740914    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cd86bdd-8465-4676-9491-2266a2e86b3a-istio-token\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g\" (UID: \"0cd86bdd-8465-4676-9491-2266a2e86b3a\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g"
Nov 03 22:28:28.741302 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:28.741000    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cd86bdd-8465-4676-9491-2266a2e86b3a-istio-podinfo\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g\" (UID: \"0cd86bdd-8465-4676-9491-2266a2e86b3a\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g"
Nov 03 22:28:28.741302 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:28.741048    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cd86bdd-8465-4676-9491-2266a2e86b3a-credential-socket\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g\" (UID: \"0cd86bdd-8465-4676-9491-2266a2e86b3a\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g"
Nov 03 22:28:28.741302 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:28.741086    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cd86bdd-8465-4676-9491-2266a2e86b3a-istiod-ca-cert\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g\" (UID: \"0cd86bdd-8465-4676-9491-2266a2e86b3a\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g"
Nov 03 22:28:28.741986 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:28.741729    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cd86bdd-8465-4676-9491-2266a2e86b3a-workload-certs\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g\" (UID: \"0cd86bdd-8465-4676-9491-2266a2e86b3a\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g"
Nov 03 22:28:28.741986 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:28.741775    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cd86bdd-8465-4676-9491-2266a2e86b3a-workload-socket\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g\" (UID: \"0cd86bdd-8465-4676-9491-2266a2e86b3a\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g"
Nov 03 22:28:28.741986 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:28.741882    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwjx4\" (UniqueName: \"kubernetes.io/projected/0cd86bdd-8465-4676-9491-2266a2e86b3a-kube-api-access-pwjx4\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g\" (UID: \"0cd86bdd-8465-4676-9491-2266a2e86b3a\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g"
Nov 03 22:28:28.741986 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:28.741908    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cd86bdd-8465-4676-9491-2266a2e86b3a-istio-data\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g\" (UID: \"0cd86bdd-8465-4676-9491-2266a2e86b3a\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g"
Nov 03 22:28:28.843473 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:28.843321    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cd86bdd-8465-4676-9491-2266a2e86b3a-istio-podinfo\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g\" (UID: \"0cd86bdd-8465-4676-9491-2266a2e86b3a\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g"
Nov 03 22:28:28.843473 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:28.843398    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cd86bdd-8465-4676-9491-2266a2e86b3a-credential-socket\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g\" (UID: \"0cd86bdd-8465-4676-9491-2266a2e86b3a\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g"
Nov 03 22:28:28.843473 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:28.843433    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cd86bdd-8465-4676-9491-2266a2e86b3a-istiod-ca-cert\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g\" (UID: \"0cd86bdd-8465-4676-9491-2266a2e86b3a\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g"
Nov 03 22:28:28.845371 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:28.843879    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cd86bdd-8465-4676-9491-2266a2e86b3a-workload-certs\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g\" (UID: \"0cd86bdd-8465-4676-9491-2266a2e86b3a\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g"
Nov 03 22:28:28.845371 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:28.843921    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cd86bdd-8465-4676-9491-2266a2e86b3a-workload-socket\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g\" (UID: \"0cd86bdd-8465-4676-9491-2266a2e86b3a\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g"
Nov 03 22:28:28.845371 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:28.844017    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwjx4\" (UniqueName: \"kubernetes.io/projected/0cd86bdd-8465-4676-9491-2266a2e86b3a-kube-api-access-pwjx4\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g\" (UID: \"0cd86bdd-8465-4676-9491-2266a2e86b3a\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g"
Nov 03 22:28:28.845371 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:28.844072    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cd86bdd-8465-4676-9491-2266a2e86b3a-istio-data\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g\" (UID: \"0cd86bdd-8465-4676-9491-2266a2e86b3a\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g"
Nov 03 22:28:28.845371 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:28.844170    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cd86bdd-8465-4676-9491-2266a2e86b3a-istio-envoy\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g\" (UID: \"0cd86bdd-8465-4676-9491-2266a2e86b3a\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g"
Nov 03 22:28:28.845371 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:28.844310    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cd86bdd-8465-4676-9491-2266a2e86b3a-istio-token\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g\" (UID: \"0cd86bdd-8465-4676-9491-2266a2e86b3a\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g"
Nov 03 22:28:28.845371 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:28.844680    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cd86bdd-8465-4676-9491-2266a2e86b3a-istiod-ca-cert\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g\" (UID: \"0cd86bdd-8465-4676-9491-2266a2e86b3a\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g"
Nov 03 22:28:28.845371 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:28.845005    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cd86bdd-8465-4676-9491-2266a2e86b3a-workload-certs\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g\" (UID: \"0cd86bdd-8465-4676-9491-2266a2e86b3a\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g"
Nov 03 22:28:28.845371 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:28.845037    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cd86bdd-8465-4676-9491-2266a2e86b3a-credential-socket\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g\" (UID: \"0cd86bdd-8465-4676-9491-2266a2e86b3a\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g"
Nov 03 22:28:28.845371 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:28.845315    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cd86bdd-8465-4676-9491-2266a2e86b3a-workload-socket\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g\" (UID: \"0cd86bdd-8465-4676-9491-2266a2e86b3a\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g"
Nov 03 22:28:28.845954 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:28.845738    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cd86bdd-8465-4676-9491-2266a2e86b3a-istio-data\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g\" (UID: \"0cd86bdd-8465-4676-9491-2266a2e86b3a\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g"
Nov 03 22:28:28.848280 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:28.848255    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cd86bdd-8465-4676-9491-2266a2e86b3a-istio-envoy\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g\" (UID: \"0cd86bdd-8465-4676-9491-2266a2e86b3a\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g"
Nov 03 22:28:28.849980 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:28.849954    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cd86bdd-8465-4676-9491-2266a2e86b3a-istio-podinfo\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g\" (UID: \"0cd86bdd-8465-4676-9491-2266a2e86b3a\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g"
Nov 03 22:28:28.869570 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:28.869495    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwjx4\" (UniqueName: \"kubernetes.io/projected/0cd86bdd-8465-4676-9491-2266a2e86b3a-kube-api-access-pwjx4\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g\" (UID: \"0cd86bdd-8465-4676-9491-2266a2e86b3a\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g"
Nov 03 22:28:28.872683 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:28.872656    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cd86bdd-8465-4676-9491-2266a2e86b3a-istio-token\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g\" (UID: \"0cd86bdd-8465-4676-9491-2266a2e86b3a\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g"
Nov 03 22:28:29.034134 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:29.033955    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g"
Nov 03 22:28:29.519240 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:29.509711    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:28:30.158269 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:30.083320    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5187e1/distinct-w3-5187e1-kserve-545b7fb6bc-mk49g" podUID="3ba8dc17-1b1b-4811-9dd6-595d75c6ca04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:28:30.158269 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:30.158195    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-50d738/premium-w4-50d738-kserve-59ccd7f995-xbfq2" podUID="fa0a680b-8f0d-4b19-8217-82282764511a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:28:30.159250 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:28:30.159192    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cd86bdd_8465_4676_9491_2266a2e86b3a.slice/crio-765eb6e4cf6372c519ef96b493bf320d8fda3da591cebbebe27873b8b59697b6 WatchSource:0}: Error finding container 765eb6e4cf6372c519ef96b493bf320d8fda3da591cebbebe27873b8b59697b6: Status 404 returned error can't find the container with id 765eb6e4cf6372c519ef96b493bf320d8fda3da591cebbebe27873b8b59697b6
Nov 03 22:28:30.162695 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:30.162663    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-50d738/distinct-w4-50d738-kserve-668fdf8b86-5pp9p" podUID="ce50af62-9cb6-4379-99c7-b40630538f4f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:28:30.230545 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:30.230494    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-553753/distinct-2-w2-553753-kserve-5bf8568844-7q4l2" podUID="a1894b80-164c-4ce1-94f3-1e5011d9759e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:28:30.254339 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:30.249697    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:28:30.254339 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:30.249801    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:28:30.254339 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:30.249843    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:28:30.351155 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:30.351057    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g"]
Nov 03 22:28:30.706395 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:30.694791    4431 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5187e1/premium-w3-5187e1-kserve-56bf8788d4-pp8st" podUID="a70ecfe9-020a-4331-b67c-aaaa46ed1eb1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": context deadline exceeded"
Nov 03 22:28:30.873630 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:30.873587    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/simulator-w0-88a4d9-kserve-785f84888f-sjbzp" podUID="18bd460b-a0f1-415c-bcb4-41c9f70d2926" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:28:31.004276 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:31.004203    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-w6-2d619f-kserve-74dbc56695-tnvtb" podUID="90310d72-c818-4fdd-b4e7-b51f633caa05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:28:31.020059 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:31.019997    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-553753/simulator-w2-553753-kserve-64d77d4dfc-4rlbn" podUID="a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Nov 03 22:28:31.032446 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:31.032398    4431 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-553753/simulator-w2-553753-kserve-64d77d4dfc-4rlbn" podUID="a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:28:31.077300 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:31.047352    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5187e1/premium-w3-5187e1-kserve-56bf8788d4-pp8st" podUID="a70ecfe9-020a-4331-b67c-aaaa46ed1eb1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:28:31.091447 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:31.091382    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5187e1/distinct-w3-5187e1-kserve-545b7fb6bc-mk49g" podUID="3ba8dc17-1b1b-4811-9dd6-595d75c6ca04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:28:31.096402 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:31.096364    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5187e1/distinct-w3-5187e1-kserve-545b7fb6bc-mk49g"
Nov 03 22:28:31.139334 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:31.135124    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/premium-w0-88a4d9-kserve-5559d75b7b-vqllq" podUID="05206737-4ca3-41fc-b35a-37c625cc182c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 03 22:28:31.147407 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:31.147364    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g" event={"ID":"0cd86bdd-8465-4676-9491-2266a2e86b3a","Type":"ContainerStarted","Data":"765eb6e4cf6372c519ef96b493bf320d8fda3da591cebbebe27873b8b59697b6"}
Nov 03 22:28:31.169698 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:31.169637    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-50d738/premium-w4-50d738-kserve-59ccd7f995-xbfq2" podUID="fa0a680b-8f0d-4b19-8217-82282764511a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:28:31.170178 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:31.170124    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-50d738/premium-w4-50d738-kserve-59ccd7f995-xbfq2"
Nov 03 22:28:31.195791 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:31.195740    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-50d738/distinct-w4-50d738-kserve-668fdf8b86-5pp9p" podUID="ce50af62-9cb6-4379-99c7-b40630538f4f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:28:31.211501 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:31.210383    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-50d738/distinct-w4-50d738-kserve-668fdf8b86-5pp9p"
Nov 03 22:28:31.260699 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:31.260651    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-553753/distinct-2-w2-553753-kserve-5bf8568844-7q4l2" podUID="a1894b80-164c-4ce1-94f3-1e5011d9759e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:28:31.261411 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:31.261385    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-553753/distinct-2-w2-553753-kserve-5bf8568844-7q4l2"
Nov 03 22:28:31.367642 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:31.367191    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-50d738/distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r" podUID="e45b1572-6519-4c71-884a-115eeddd86cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:28:31.384185 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:31.384150    4431 patch_prober.go:29] interesting pod/e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 22:28:31.384452 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:31.384427    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96" podUID="b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:28:31.522241 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:31.506718    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-50d738/distinct-w4-50d738-kserve-668fdf8b86-5pp9p"
Nov 03 22:28:31.836475 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:28:31.817877    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:28:31.884739 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:31.884689    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/simulator-w0-88a4d9-kserve-785f84888f-sjbzp" podUID="18bd460b-a0f1-415c-bcb4-41c9f70d2926" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:28:31.908240 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:31.885934    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-88a4d9/simulator-w0-88a4d9-kserve-785f84888f-sjbzp"
Nov 03 22:28:32.031383 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:32.031331    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-553753/simulator-w2-553753-kserve-64d77d4dfc-4rlbn" podUID="a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:28:32.047245 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:32.032372    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-553753/simulator-w2-553753-kserve-64d77d4dfc-4rlbn"
Nov 03 22:28:32.072821 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:32.046332    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-w6-2d619f-kserve-74dbc56695-tnvtb" podUID="90310d72-c818-4fdd-b4e7-b51f633caa05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 03 22:28:32.074003 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:32.048480    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5187e1/premium-w3-5187e1-kserve-56bf8788d4-pp8st" podUID="a70ecfe9-020a-4331-b67c-aaaa46ed1eb1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:28:32.075351 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:32.075331    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2d619f/distinct-w6-2d619f-kserve-74dbc56695-tnvtb"
Nov 03 22:28:32.075537 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:32.075523    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5187e1/premium-w3-5187e1-kserve-56bf8788d4-pp8st"
Nov 03 22:28:32.158280 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:32.157309    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5187e1/distinct-w3-5187e1-kserve-545b7fb6bc-mk49g" podUID="3ba8dc17-1b1b-4811-9dd6-595d75c6ca04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Nov 03 22:28:32.267239 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:32.225839    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-50d738/premium-w4-50d738-kserve-59ccd7f995-xbfq2" podUID="fa0a680b-8f0d-4b19-8217-82282764511a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 03 22:28:32.269428 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:32.268496    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-553753/distinct-2-w2-553753-kserve-5bf8568844-7q4l2" podUID="a1894b80-164c-4ce1-94f3-1e5011d9759e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:28:32.377240 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:32.355633    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-553753/simulator-w2-553753-kserve-64d77d4dfc-4rlbn"
Nov 03 22:28:32.400294 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:32.377581    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-50d738/distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r" podUID="e45b1572-6519-4c71-884a-115eeddd86cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Nov 03 22:28:32.400294 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:32.377945    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-50d738/distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r"
Nov 03 22:28:32.400294 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:32.388675    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g" event={"ID":"0cd86bdd-8465-4676-9491-2266a2e86b3a","Type":"ContainerStarted","Data":"7c805bc5f4e30bc755a2ea2fd94a1b6bbb3af20626acabc892dfd21ee7c6879a"}
Nov 03 22:28:32.729239 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:32.726145    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-50d738/distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r"
Nov 03 22:28:32.729239 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:32.728333    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:28:32.901486 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:32.901441    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/simulator-w0-88a4d9-kserve-785f84888f-sjbzp" podUID="18bd460b-a0f1-415c-bcb4-41c9f70d2926" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded"
Nov 03 22:28:33.055297 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:33.051143    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g"
Nov 03 22:28:33.091239 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:33.066447    4431 patch_prober.go:29] interesting pod/e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 03 22:28:33.091540 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:33.091504    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g" podUID="0cd86bdd-8465-4676-9491-2266a2e86b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 03 22:28:33.142624 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:33.093363    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-w6-2d619f-kserve-74dbc56695-tnvtb" podUID="90310d72-c818-4fdd-b4e7-b51f633caa05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:28:33.262398 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:33.179087    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5187e1/distinct-w3-5187e1-kserve-545b7fb6bc-mk49g" podUID="3ba8dc17-1b1b-4811-9dd6-595d75c6ca04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:28:33.265154 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:33.227869    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-50d738/premium-w4-50d738-kserve-59ccd7f995-xbfq2" podUID="fa0a680b-8f0d-4b19-8217-82282764511a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:28:33.265992 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:33.255503    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-88a4d9/simulator-w0-88a4d9-kserve-785f84888f-sjbzp"
Nov 03 22:28:33.267275 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:33.255786    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5187e1/premium-w3-5187e1-kserve-56bf8788d4-pp8st" podUID="a70ecfe9-020a-4331-b67c-aaaa46ed1eb1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:28:33.268748 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:33.268722    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-553753/distinct-2-w2-553753-kserve-5bf8568844-7q4l2" podUID="a1894b80-164c-4ce1-94f3-1e5011d9759e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:28:33.303060 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:33.303017    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-553753/distinct-2-w2-553753-kserve-5bf8568844-7q4l2"
Nov 03 22:28:33.527668 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:33.524628    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5187e1/premium-w3-5187e1-kserve-56bf8788d4-pp8st"
Nov 03 22:28:33.534186 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:33.534159    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5187e1/distinct-w3-5187e1-kserve-545b7fb6bc-mk49g"
Nov 03 22:28:33.587464 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:33.584854    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-50d738/premium-w4-50d738-kserve-59ccd7f995-xbfq2"
Nov 03 22:28:33.589890 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:33.589857    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2d619f/distinct-w6-2d619f-kserve-74dbc56695-tnvtb"
Nov 03 22:28:33.599876 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:33.599823    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g" podStartSLOduration=5.59980466 podStartE2EDuration="5.59980466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:28:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:28:33.021726745 +0000 UTC m=+937.047582610" watchObservedRunningTime="2026-11-03 22:28:33.59980466 +0000 UTC m=+937.625660523"
Nov 03 22:28:34.035008 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:34.034970    4431 patch_prober.go:29] interesting pod/e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 03 22:28:34.035191 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:34.035027    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g" podUID="0cd86bdd-8465-4676-9491-2266a2e86b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 03 22:28:34.461240 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:34.459699    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:28:35.035010 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:35.034971    4431 patch_prober.go:29] interesting pod/e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 03 22:28:35.035164 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:35.035025    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g" podUID="0cd86bdd-8465-4676-9491-2266a2e86b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 03 22:28:36.034747 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:36.034713    4431 patch_prober.go:29] interesting pod/e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 03 22:28:36.035132 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:36.034766    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g" podUID="0cd86bdd-8465-4676-9491-2266a2e86b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 03 22:28:36.218680 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:36.218648    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:28:37.034917 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:37.034881    4431 patch_prober.go:29] interesting pod/e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 03 22:28:37.035342 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:37.034941    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g" podUID="0cd86bdd-8465-4676-9491-2266a2e86b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 03 22:28:37.803096 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:37.803064    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:28:38.035168 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:38.035133    4431 patch_prober.go:29] interesting pod/e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 03 22:28:38.035663 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:38.035191    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g" podUID="0cd86bdd-8465-4676-9491-2266a2e86b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 03 22:28:39.034665 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:39.034623    4431 patch_prober.go:29] interesting pod/e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 03 22:28:39.034827 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:39.034694    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g" podUID="0cd86bdd-8465-4676-9491-2266a2e86b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 03 22:28:39.295043 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:39.294963    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:28:40.039794 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:40.039766    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g"
Nov 03 22:28:40.040071 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:40.040039    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g"
Nov 03 22:28:40.040934 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:40.040916    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g"
Nov 03 22:28:40.901163 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:40.901117    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:28:42.507913 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:42.507879    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:28:43.582288 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:43.582248    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5"]
Nov 03 22:28:43.587632 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:43.587604    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5"
Nov 03 22:28:43.621413 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:43.621380    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5"]
Nov 03 22:28:43.750111 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:43.750080    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb2c0928-23fa-462f-baa4-4bd63dab4d62-workload-certs\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5\" (UID: \"bb2c0928-23fa-462f-baa4-4bd63dab4d62\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5"
Nov 03 22:28:43.750305 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:43.750125    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb2c0928-23fa-462f-baa4-4bd63dab4d62-istio-token\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5\" (UID: \"bb2c0928-23fa-462f-baa4-4bd63dab4d62\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5"
Nov 03 22:28:43.750305 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:43.750257    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb2c0928-23fa-462f-baa4-4bd63dab4d62-istio-envoy\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5\" (UID: \"bb2c0928-23fa-462f-baa4-4bd63dab4d62\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5"
Nov 03 22:28:43.750393 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:43.750306    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb2c0928-23fa-462f-baa4-4bd63dab4d62-istiod-ca-cert\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5\" (UID: \"bb2c0928-23fa-462f-baa4-4bd63dab4d62\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5"
Nov 03 22:28:43.750393 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:43.750369    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb2c0928-23fa-462f-baa4-4bd63dab4d62-credential-socket\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5\" (UID: \"bb2c0928-23fa-462f-baa4-4bd63dab4d62\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5"
Nov 03 22:28:43.750460 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:43.750430    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pk48z\" (UniqueName: \"kubernetes.io/projected/bb2c0928-23fa-462f-baa4-4bd63dab4d62-kube-api-access-pk48z\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5\" (UID: \"bb2c0928-23fa-462f-baa4-4bd63dab4d62\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5"
Nov 03 22:28:43.750460 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:43.750451    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb2c0928-23fa-462f-baa4-4bd63dab4d62-workload-socket\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5\" (UID: \"bb2c0928-23fa-462f-baa4-4bd63dab4d62\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5"
Nov 03 22:28:43.750521 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:43.750476    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb2c0928-23fa-462f-baa4-4bd63dab4d62-istio-podinfo\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5\" (UID: \"bb2c0928-23fa-462f-baa4-4bd63dab4d62\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5"
Nov 03 22:28:43.750551 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:43.750522    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb2c0928-23fa-462f-baa4-4bd63dab4d62-istio-data\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5\" (UID: \"bb2c0928-23fa-462f-baa4-4bd63dab4d62\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5"
Nov 03 22:28:43.852352 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:43.852268    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb2c0928-23fa-462f-baa4-4bd63dab4d62-istio-envoy\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5\" (UID: \"bb2c0928-23fa-462f-baa4-4bd63dab4d62\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5"
Nov 03 22:28:43.852352 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:43.852327    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb2c0928-23fa-462f-baa4-4bd63dab4d62-istiod-ca-cert\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5\" (UID: \"bb2c0928-23fa-462f-baa4-4bd63dab4d62\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5"
Nov 03 22:28:43.852608 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:43.852358    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb2c0928-23fa-462f-baa4-4bd63dab4d62-credential-socket\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5\" (UID: \"bb2c0928-23fa-462f-baa4-4bd63dab4d62\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5"
Nov 03 22:28:43.852608 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:43.852400    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pk48z\" (UniqueName: \"kubernetes.io/projected/bb2c0928-23fa-462f-baa4-4bd63dab4d62-kube-api-access-pk48z\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5\" (UID: \"bb2c0928-23fa-462f-baa4-4bd63dab4d62\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5"
Nov 03 22:28:43.852608 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:43.852428    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb2c0928-23fa-462f-baa4-4bd63dab4d62-workload-socket\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5\" (UID: \"bb2c0928-23fa-462f-baa4-4bd63dab4d62\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5"
Nov 03 22:28:43.852608 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:43.852453    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb2c0928-23fa-462f-baa4-4bd63dab4d62-istio-podinfo\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5\" (UID: \"bb2c0928-23fa-462f-baa4-4bd63dab4d62\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5"
Nov 03 22:28:43.852608 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:43.852494    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb2c0928-23fa-462f-baa4-4bd63dab4d62-istio-data\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5\" (UID: \"bb2c0928-23fa-462f-baa4-4bd63dab4d62\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5"
Nov 03 22:28:43.852868 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:43.852695    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb2c0928-23fa-462f-baa4-4bd63dab4d62-workload-certs\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5\" (UID: \"bb2c0928-23fa-462f-baa4-4bd63dab4d62\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5"
Nov 03 22:28:43.852868 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:43.852747    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb2c0928-23fa-462f-baa4-4bd63dab4d62-istio-token\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5\" (UID: \"bb2c0928-23fa-462f-baa4-4bd63dab4d62\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5"
Nov 03 22:28:43.852868 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:43.852846    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb2c0928-23fa-462f-baa4-4bd63dab4d62-credential-socket\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5\" (UID: \"bb2c0928-23fa-462f-baa4-4bd63dab4d62\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5"
Nov 03 22:28:43.853018 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:43.852912    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb2c0928-23fa-462f-baa4-4bd63dab4d62-workload-socket\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5\" (UID: \"bb2c0928-23fa-462f-baa4-4bd63dab4d62\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5"
Nov 03 22:28:43.853094 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:43.853073    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb2c0928-23fa-462f-baa4-4bd63dab4d62-workload-certs\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5\" (UID: \"bb2c0928-23fa-462f-baa4-4bd63dab4d62\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5"
Nov 03 22:28:43.853245 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:43.853195    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb2c0928-23fa-462f-baa4-4bd63dab4d62-istiod-ca-cert\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5\" (UID: \"bb2c0928-23fa-462f-baa4-4bd63dab4d62\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5"
Nov 03 22:28:43.853366 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:43.853295    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb2c0928-23fa-462f-baa4-4bd63dab4d62-istio-data\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5\" (UID: \"bb2c0928-23fa-462f-baa4-4bd63dab4d62\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5"
Nov 03 22:28:43.855135 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:43.855112    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb2c0928-23fa-462f-baa4-4bd63dab4d62-istio-envoy\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5\" (UID: \"bb2c0928-23fa-462f-baa4-4bd63dab4d62\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5"
Nov 03 22:28:43.855428 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:43.855406    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb2c0928-23fa-462f-baa4-4bd63dab4d62-istio-podinfo\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5\" (UID: \"bb2c0928-23fa-462f-baa4-4bd63dab4d62\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5"
Nov 03 22:28:43.898304 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:43.898271    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb2c0928-23fa-462f-baa4-4bd63dab4d62-istio-token\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5\" (UID: \"bb2c0928-23fa-462f-baa4-4bd63dab4d62\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5"
Nov 03 22:28:43.898587 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:43.898563    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pk48z\" (UniqueName: \"kubernetes.io/projected/bb2c0928-23fa-462f-baa4-4bd63dab4d62-kube-api-access-pk48z\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5\" (UID: \"bb2c0928-23fa-462f-baa4-4bd63dab4d62\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5"
Nov 03 22:28:43.907635 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:43.907607    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5"
Nov 03 22:28:44.164139 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:44.164100    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf"]
Nov 03 22:28:44.169514 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:44.169488    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf"
Nov 03 22:28:44.192350 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:44.192271    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf"]
Nov 03 22:28:44.256526 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:44.256503    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5"]
Nov 03 22:28:44.258174 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:28:44.258149    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb2c0928_23fa_462f_baa4_4bd63dab4d62.slice/crio-0a82386da5b90be5b6e1b671a757a3c46daf57cedf1217557f7073a745b17bb7 WatchSource:0}: Error finding container 0a82386da5b90be5b6e1b671a757a3c46daf57cedf1217557f7073a745b17bb7: Status 404 returned error can't find the container with id 0a82386da5b90be5b6e1b671a757a3c46daf57cedf1217557f7073a745b17bb7
Nov 03 22:28:44.260659 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:44.260626    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:28:44.260761 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:44.260701    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:28:44.260828 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:44.260773    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:28:44.298005 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:44.297970    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:28:44.358447 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:44.358421    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4db918b5-a69e-45f7-996a-2658132cbffc-workload-certs\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf\" (UID: \"4db918b5-a69e-45f7-996a-2658132cbffc\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf"
Nov 03 22:28:44.358584 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:44.358466    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4db918b5-a69e-45f7-996a-2658132cbffc-istiod-ca-cert\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf\" (UID: \"4db918b5-a69e-45f7-996a-2658132cbffc\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf"
Nov 03 22:28:44.358584 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:44.358557    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4db918b5-a69e-45f7-996a-2658132cbffc-istio-envoy\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf\" (UID: \"4db918b5-a69e-45f7-996a-2658132cbffc\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf"
Nov 03 22:28:44.358785 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:44.358767    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4db918b5-a69e-45f7-996a-2658132cbffc-istio-data\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf\" (UID: \"4db918b5-a69e-45f7-996a-2658132cbffc\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf"
Nov 03 22:28:44.358893 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:44.358880    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4db918b5-a69e-45f7-996a-2658132cbffc-istio-token\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf\" (UID: \"4db918b5-a69e-45f7-996a-2658132cbffc\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf"
Nov 03 22:28:44.358932 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:44.358905    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4db918b5-a69e-45f7-996a-2658132cbffc-workload-socket\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf\" (UID: \"4db918b5-a69e-45f7-996a-2658132cbffc\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf"
Nov 03 22:28:44.358932 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:44.358922    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw7h5\" (UniqueName: \"kubernetes.io/projected/4db918b5-a69e-45f7-996a-2658132cbffc-kube-api-access-zw7h5\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf\" (UID: \"4db918b5-a69e-45f7-996a-2658132cbffc\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf"
Nov 03 22:28:44.359011 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:44.358956    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4db918b5-a69e-45f7-996a-2658132cbffc-istio-podinfo\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf\" (UID: \"4db918b5-a69e-45f7-996a-2658132cbffc\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf"
Nov 03 22:28:44.359046 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:44.359025    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4db918b5-a69e-45f7-996a-2658132cbffc-credential-socket\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf\" (UID: \"4db918b5-a69e-45f7-996a-2658132cbffc\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf"
Nov 03 22:28:44.460589 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:44.460505    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4db918b5-a69e-45f7-996a-2658132cbffc-istio-data\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf\" (UID: \"4db918b5-a69e-45f7-996a-2658132cbffc\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf"
Nov 03 22:28:44.460761 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:44.460616    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4db918b5-a69e-45f7-996a-2658132cbffc-istio-token\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf\" (UID: \"4db918b5-a69e-45f7-996a-2658132cbffc\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf"
Nov 03 22:28:44.460761 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:44.460647    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4db918b5-a69e-45f7-996a-2658132cbffc-workload-socket\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf\" (UID: \"4db918b5-a69e-45f7-996a-2658132cbffc\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf"
Nov 03 22:28:44.460761 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:44.460671    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw7h5\" (UniqueName: \"kubernetes.io/projected/4db918b5-a69e-45f7-996a-2658132cbffc-kube-api-access-zw7h5\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf\" (UID: \"4db918b5-a69e-45f7-996a-2658132cbffc\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf"
Nov 03 22:28:44.460761 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:44.460711    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4db918b5-a69e-45f7-996a-2658132cbffc-istio-podinfo\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf\" (UID: \"4db918b5-a69e-45f7-996a-2658132cbffc\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf"
Nov 03 22:28:44.460971 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:44.460827    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4db918b5-a69e-45f7-996a-2658132cbffc-credential-socket\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf\" (UID: \"4db918b5-a69e-45f7-996a-2658132cbffc\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf"
Nov 03 22:28:44.460971 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:44.460902    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4db918b5-a69e-45f7-996a-2658132cbffc-workload-certs\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf\" (UID: \"4db918b5-a69e-45f7-996a-2658132cbffc\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf"
Nov 03 22:28:44.460971 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:44.460949    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4db918b5-a69e-45f7-996a-2658132cbffc-istiod-ca-cert\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf\" (UID: \"4db918b5-a69e-45f7-996a-2658132cbffc\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf"
Nov 03 22:28:44.461138 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:44.460985    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4db918b5-a69e-45f7-996a-2658132cbffc-istio-envoy\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf\" (UID: \"4db918b5-a69e-45f7-996a-2658132cbffc\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf"
Nov 03 22:28:44.461138 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:44.461034    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4db918b5-a69e-45f7-996a-2658132cbffc-istio-data\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf\" (UID: \"4db918b5-a69e-45f7-996a-2658132cbffc\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf"
Nov 03 22:28:44.461138 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:44.461113    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4db918b5-a69e-45f7-996a-2658132cbffc-workload-socket\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf\" (UID: \"4db918b5-a69e-45f7-996a-2658132cbffc\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf"
Nov 03 22:28:44.461339 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:44.461312    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4db918b5-a69e-45f7-996a-2658132cbffc-credential-socket\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf\" (UID: \"4db918b5-a69e-45f7-996a-2658132cbffc\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf"
Nov 03 22:28:44.461404 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:44.461387    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4db918b5-a69e-45f7-996a-2658132cbffc-workload-certs\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf\" (UID: \"4db918b5-a69e-45f7-996a-2658132cbffc\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf"
Nov 03 22:28:44.461644 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:44.461621    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4db918b5-a69e-45f7-996a-2658132cbffc-istiod-ca-cert\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf\" (UID: \"4db918b5-a69e-45f7-996a-2658132cbffc\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf"
Nov 03 22:28:44.463272 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:44.463250    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4db918b5-a69e-45f7-996a-2658132cbffc-istio-envoy\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf\" (UID: \"4db918b5-a69e-45f7-996a-2658132cbffc\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf"
Nov 03 22:28:44.463359 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:44.463288    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4db918b5-a69e-45f7-996a-2658132cbffc-istio-podinfo\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf\" (UID: \"4db918b5-a69e-45f7-996a-2658132cbffc\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf"
Nov 03 22:28:44.491088 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:44.491060    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4db918b5-a69e-45f7-996a-2658132cbffc-istio-token\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf\" (UID: \"4db918b5-a69e-45f7-996a-2658132cbffc\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf"
Nov 03 22:28:44.491292 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:44.491139    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw7h5\" (UniqueName: \"kubernetes.io/projected/4db918b5-a69e-45f7-996a-2658132cbffc-kube-api-access-zw7h5\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf\" (UID: \"4db918b5-a69e-45f7-996a-2658132cbffc\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf"
Nov 03 22:28:44.575672 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:44.575635    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5" event={"ID":"bb2c0928-23fa-462f-baa4-4bd63dab4d62","Type":"ContainerStarted","Data":"bd14f1cdfdad496f626af92146563f497107c9b8015311eca462a239d0cc1d2d"}
Nov 03 22:28:44.575837 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:44.575686    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5" event={"ID":"bb2c0928-23fa-462f-baa4-4bd63dab4d62","Type":"ContainerStarted","Data":"0a82386da5b90be5b6e1b671a757a3c46daf57cedf1217557f7073a745b17bb7"}
Nov 03 22:28:44.620097 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:44.620041    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5" podStartSLOduration=1.620028809 podStartE2EDuration="1.620028809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:28:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:28:44.61891187 +0000 UTC m=+948.644767732" watchObservedRunningTime="2026-11-03 22:28:44.620028809 +0000 UTC m=+948.645884671"
Nov 03 22:28:44.786109 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:44.786020    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf"
Nov 03 22:28:44.908017 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:44.907980    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5"
Nov 03 22:28:45.111279 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:45.111246    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf"]
Nov 03 22:28:45.112049 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:28:45.112025    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4db918b5_a69e_45f7_996a_2658132cbffc.slice/crio-e2d1d2e2da42f58fc773edb02e6becb6bfb85016f04c7e8e9aebf71f55864cc1 WatchSource:0}: Error finding container e2d1d2e2da42f58fc773edb02e6becb6bfb85016f04c7e8e9aebf71f55864cc1: Status 404 returned error can't find the container with id e2d1d2e2da42f58fc773edb02e6becb6bfb85016f04c7e8e9aebf71f55864cc1
Nov 03 22:28:45.114600 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:45.114562    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:28:45.114708 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:45.114627    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:28:45.114708 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:45.114656    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:28:45.528340 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:45.528299    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:28:45.584653 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:45.584611    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf" event={"ID":"4db918b5-a69e-45f7-996a-2658132cbffc","Type":"ContainerStarted","Data":"3e2220b8c700b90cf1c7eff5571b93b90a3c2f6031f4c061ca140100387d0e73"}
Nov 03 22:28:45.584873 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:45.584671    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf" event={"ID":"4db918b5-a69e-45f7-996a-2658132cbffc","Type":"ContainerStarted","Data":"e2d1d2e2da42f58fc773edb02e6becb6bfb85016f04c7e8e9aebf71f55864cc1"}
Nov 03 22:28:45.714916 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:45.713327    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:28:45.790302 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:45.789320    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf"
Nov 03 22:28:45.878242 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:45.866585    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:28:45.911762 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:45.909685    4431 patch_prober.go:29] interesting pod/e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 03 22:28:45.911762 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:45.909749    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5" podUID="bb2c0928-23fa-462f-baa4-4bd63dab4d62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 03 22:28:46.023578 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:46.023543    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:28:46.276246 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:46.265904    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:28:46.413166 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:46.413125    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:28:46.790610 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:46.790579    4431 patch_prober.go:29] interesting pod/e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 03 22:28:46.791074 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:46.791047    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf" podUID="4db918b5-a69e-45f7-996a-2658132cbffc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 03 22:28:46.908482 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:46.908445    4431 patch_prober.go:29] interesting pod/e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 03 22:28:46.908684 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:46.908507    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5" podUID="bb2c0928-23fa-462f-baa4-4bd63dab4d62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 03 22:28:47.795378 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:47.795340    4431 patch_prober.go:29] interesting pod/e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 03 22:28:47.795876 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:47.795392    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf" podUID="4db918b5-a69e-45f7-996a-2658132cbffc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 03 22:28:47.908110 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:47.908074    4431 patch_prober.go:29] interesting pod/e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 03 22:28:47.908301 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:47.908127    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5" podUID="bb2c0928-23fa-462f-baa4-4bd63dab4d62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 03 22:28:48.052293 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:48.051643    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:28:48.789725 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:48.789686    4431 patch_prober.go:29] interesting pod/e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 03 22:28:48.789926 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:48.789762    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf" podUID="4db918b5-a69e-45f7-996a-2658132cbffc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 03 22:28:48.806487 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:48.806428    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:28:48.908192 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:48.908154    4431 patch_prober.go:29] interesting pod/e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 03 22:28:48.908377 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:48.908234    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5" podUID="bb2c0928-23fa-462f-baa4-4bd63dab4d62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 03 22:28:48.971239 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:48.970630    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:28:49.129780 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:49.129663    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:28:49.720241 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:49.711858    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:28:49.798240 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:49.787875    4431 patch_prober.go:29] interesting pod/e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 03 22:28:49.798240 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:49.787949    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf" podUID="4db918b5-a69e-45f7-996a-2658132cbffc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 03 22:28:49.918274 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:49.917114    4431 patch_prober.go:29] interesting pod/e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 03 22:28:49.918274 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:49.917198    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5" podUID="bb2c0928-23fa-462f-baa4-4bd63dab4d62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 03 22:28:49.923366 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:49.923328    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:28:50.134381 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:50.134271    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:28:50.619516 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:50.619480    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:28:50.787588 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:50.787551    4431 patch_prober.go:29] interesting pod/e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 03 22:28:50.787773 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:50.787621    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf" podUID="4db918b5-a69e-45f7-996a-2658132cbffc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 03 22:28:50.908299 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:50.908259    4431 patch_prober.go:29] interesting pod/e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 03 22:28:50.908451 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:50.908326    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5" podUID="bb2c0928-23fa-462f-baa4-4bd63dab4d62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 03 22:28:51.786575 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:51.786543    4431 patch_prober.go:29] interesting pod/e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 03 22:28:51.787202 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:51.787158    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf" podUID="4db918b5-a69e-45f7-996a-2658132cbffc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 03 22:28:51.836005 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:51.835920    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:28:52.105902 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:52.105813    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5"
Nov 03 22:28:52.106162 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:52.106142    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5"
Nov 03 22:28:52.107000 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:52.106976    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5"
Nov 03 22:28:52.168165 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:52.168080    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf" podStartSLOduration=8.168052974 podStartE2EDuration="8.168052974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:28:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:28:45.66556218 +0000 UTC m=+949.691418044" watchObservedRunningTime="2026-11-03 22:28:52.168052974 +0000 UTC m=+956.193908837"
Nov 03 22:28:52.816691 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:52.816661    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf"
Nov 03 22:28:52.817135 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:52.817116    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf"
Nov 03 22:28:52.818933 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:52.818874    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf"
Nov 03 22:28:52.968393 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:52.968360    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:28:53.112552 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:53.112458    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:28:53.263665 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:53.263620    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:28:53.565943 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:53.565908    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:28:53.730927 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:53.730875    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:28:53.881344 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:53.881243    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:28:55.583063 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:55.583028    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:28:55.801085 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:55.801049    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:28:55.942531 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:55.942499    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:28:57.376623 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:57.376592    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:28:57.577981 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:57.577946    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:28:57.745309 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:57.745276    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:28:59.124132 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:59.124092    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:28:59.302448 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:59.302414    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm"]
Nov 03 22:28:59.308063 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:59.308028    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm"
Nov 03 22:28:59.330050 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:59.330008    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm"]
Nov 03 22:28:59.358990 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:59.358955    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b0aaae7-2a00-4487-a054-a359353bf4a9-istio-podinfo\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm\" (UID: \"9b0aaae7-2a00-4487-a054-a359353bf4a9\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm"
Nov 03 22:28:59.359152 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:59.359029    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b0aaae7-2a00-4487-a054-a359353bf4a9-workload-socket\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm\" (UID: \"9b0aaae7-2a00-4487-a054-a359353bf4a9\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm"
Nov 03 22:28:59.359195 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:59.359149    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b0aaae7-2a00-4487-a054-a359353bf4a9-istio-token\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm\" (UID: \"9b0aaae7-2a00-4487-a054-a359353bf4a9\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm"
Nov 03 22:28:59.359257 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:59.359202    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b0aaae7-2a00-4487-a054-a359353bf4a9-istio-envoy\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm\" (UID: \"9b0aaae7-2a00-4487-a054-a359353bf4a9\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm"
Nov 03 22:28:59.359327 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:59.359309    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b0aaae7-2a00-4487-a054-a359353bf4a9-workload-certs\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm\" (UID: \"9b0aaae7-2a00-4487-a054-a359353bf4a9\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm"
Nov 03 22:28:59.359397 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:59.359379    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b0aaae7-2a00-4487-a054-a359353bf4a9-credential-socket\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm\" (UID: \"9b0aaae7-2a00-4487-a054-a359353bf4a9\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm"
Nov 03 22:28:59.359470 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:59.359448    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b0aaae7-2a00-4487-a054-a359353bf4a9-istiod-ca-cert\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm\" (UID: \"9b0aaae7-2a00-4487-a054-a359353bf4a9\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm"
Nov 03 22:28:59.359549 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:59.359535    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-545tj\" (UniqueName: \"kubernetes.io/projected/9b0aaae7-2a00-4487-a054-a359353bf4a9-kube-api-access-545tj\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm\" (UID: \"9b0aaae7-2a00-4487-a054-a359353bf4a9\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm"
Nov 03 22:28:59.359677 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:59.359658    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b0aaae7-2a00-4487-a054-a359353bf4a9-istio-data\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm\" (UID: \"9b0aaae7-2a00-4487-a054-a359353bf4a9\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm"
Nov 03 22:28:59.461116 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:59.461083    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b0aaae7-2a00-4487-a054-a359353bf4a9-workload-certs\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm\" (UID: \"9b0aaae7-2a00-4487-a054-a359353bf4a9\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm"
Nov 03 22:28:59.461333 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:59.461143    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b0aaae7-2a00-4487-a054-a359353bf4a9-credential-socket\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm\" (UID: \"9b0aaae7-2a00-4487-a054-a359353bf4a9\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm"
Nov 03 22:28:59.461333 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:59.461257    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b0aaae7-2a00-4487-a054-a359353bf4a9-istiod-ca-cert\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm\" (UID: \"9b0aaae7-2a00-4487-a054-a359353bf4a9\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm"
Nov 03 22:28:59.461333 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:59.461323    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-545tj\" (UniqueName: \"kubernetes.io/projected/9b0aaae7-2a00-4487-a054-a359353bf4a9-kube-api-access-545tj\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm\" (UID: \"9b0aaae7-2a00-4487-a054-a359353bf4a9\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm"
Nov 03 22:28:59.461498 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:59.461463    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b0aaae7-2a00-4487-a054-a359353bf4a9-istio-data\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm\" (UID: \"9b0aaae7-2a00-4487-a054-a359353bf4a9\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm"
Nov 03 22:28:59.461580 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:59.461557    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b0aaae7-2a00-4487-a054-a359353bf4a9-istio-podinfo\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm\" (UID: \"9b0aaae7-2a00-4487-a054-a359353bf4a9\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm"
Nov 03 22:28:59.461580 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:59.461576    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b0aaae7-2a00-4487-a054-a359353bf4a9-credential-socket\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm\" (UID: \"9b0aaae7-2a00-4487-a054-a359353bf4a9\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm"
Nov 03 22:28:59.461779 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:59.461640    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b0aaae7-2a00-4487-a054-a359353bf4a9-workload-certs\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm\" (UID: \"9b0aaae7-2a00-4487-a054-a359353bf4a9\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm"
Nov 03 22:28:59.461779 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:59.461696    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b0aaae7-2a00-4487-a054-a359353bf4a9-workload-socket\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm\" (UID: \"9b0aaae7-2a00-4487-a054-a359353bf4a9\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm"
Nov 03 22:28:59.461779 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:59.461757    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b0aaae7-2a00-4487-a054-a359353bf4a9-istio-token\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm\" (UID: \"9b0aaae7-2a00-4487-a054-a359353bf4a9\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm"
Nov 03 22:28:59.461956 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:59.461798    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b0aaae7-2a00-4487-a054-a359353bf4a9-istio-envoy\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm\" (UID: \"9b0aaae7-2a00-4487-a054-a359353bf4a9\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm"
Nov 03 22:28:59.461956 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:59.461856    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b0aaae7-2a00-4487-a054-a359353bf4a9-istio-data\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm\" (UID: \"9b0aaae7-2a00-4487-a054-a359353bf4a9\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm"
Nov 03 22:28:59.462105 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:59.462083    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b0aaae7-2a00-4487-a054-a359353bf4a9-workload-socket\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm\" (UID: \"9b0aaae7-2a00-4487-a054-a359353bf4a9\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm"
Nov 03 22:28:59.462171 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:59.462090    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b0aaae7-2a00-4487-a054-a359353bf4a9-istiod-ca-cert\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm\" (UID: \"9b0aaae7-2a00-4487-a054-a359353bf4a9\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm"
Nov 03 22:28:59.463926 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:59.463902    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b0aaae7-2a00-4487-a054-a359353bf4a9-istio-envoy\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm\" (UID: \"9b0aaae7-2a00-4487-a054-a359353bf4a9\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm"
Nov 03 22:28:59.464131 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:59.464116    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b0aaae7-2a00-4487-a054-a359353bf4a9-istio-podinfo\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm\" (UID: \"9b0aaae7-2a00-4487-a054-a359353bf4a9\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm"
Nov 03 22:28:59.485453 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:59.485424    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b0aaae7-2a00-4487-a054-a359353bf4a9-istio-token\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm\" (UID: \"9b0aaae7-2a00-4487-a054-a359353bf4a9\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm"
Nov 03 22:28:59.485576 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:59.485541    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-545tj\" (UniqueName: \"kubernetes.io/projected/9b0aaae7-2a00-4487-a054-a359353bf4a9-kube-api-access-545tj\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm\" (UID: \"9b0aaae7-2a00-4487-a054-a359353bf4a9\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm"
Nov 03 22:28:59.628747 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:59.628719    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm"
Nov 03 22:28:59.915362 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:59.915332    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:28:59.955409 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:28:59.955368    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b0aaae7_2a00_4487_a054_a359353bf4a9.slice/crio-70ee89ccd97634d8de551e74d8de8e2719fdbaf1773b3232f3f2ec5d82f67fa2 WatchSource:0}: Error finding container 70ee89ccd97634d8de551e74d8de8e2719fdbaf1773b3232f3f2ec5d82f67fa2: Status 404 returned error can't find the container with id 70ee89ccd97634d8de551e74d8de8e2719fdbaf1773b3232f3f2ec5d82f67fa2
Nov 03 22:28:59.955974 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:59.955950    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm"]
Nov 03 22:28:59.958731 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:59.958690    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:28:59.958833 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:59.958779    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:28:59.958888 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:28:59.958832    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:29:00.075955 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:00.075903    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:29:00.228781 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:00.228678    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:29:00.703121 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:00.703081    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm" event={"ID":"9b0aaae7-2a00-4487-a054-a359353bf4a9","Type":"ContainerStarted","Data":"1ba7b1aca193f6a8a973a82c01b6a19130b89e1f299d752d6e70676b6caf08a9"}
Nov 03 22:29:00.703332 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:00.703147    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm" event={"ID":"9b0aaae7-2a00-4487-a054-a359353bf4a9","Type":"ContainerStarted","Data":"70ee89ccd97634d8de551e74d8de8e2719fdbaf1773b3232f3f2ec5d82f67fa2"}
Nov 03 22:29:00.764251 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:00.763882    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm" podStartSLOduration=1.763860776 podStartE2EDuration="1.763860776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:28:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:29:00.763758686 +0000 UTC m=+964.789614548" watchObservedRunningTime="2026-11-03 22:29:00.763860776 +0000 UTC m=+964.789716639"
Nov 03 22:29:01.264241 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:01.262729    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:29:01.290824 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:29:01.290783    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:29:01.519864 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:01.519761    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:29:01.629139 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:01.629105    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm"
Nov 03 22:29:01.630771 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:01.630748    4431 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 03 22:29:01.630837 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:01.630806    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm" podUID="9b0aaae7-2a00-4487-a054-a359353bf4a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 03 22:29:01.686163 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:01.686129    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:29:02.629302 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:02.629265    4431 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 03 22:29:02.629762 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:02.629337    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm" podUID="9b0aaae7-2a00-4487-a054-a359353bf4a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 03 22:29:03.436028 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:03.435997    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:29:03.629133 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:03.629099    4431 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 03 22:29:03.629350 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:03.629160    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm" podUID="9b0aaae7-2a00-4487-a054-a359353bf4a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 03 22:29:04.629344 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:04.629287    4431 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 03 22:29:04.629752 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:04.629364    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm" podUID="9b0aaae7-2a00-4487-a054-a359353bf4a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 03 22:29:05.629798 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:05.629764    4431 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 03 22:29:05.630254 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:05.629820    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm" podUID="9b0aaae7-2a00-4487-a054-a359353bf4a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 03 22:29:05.811395 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:05.811353    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:29:06.629436 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:06.629404    4431 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 03 22:29:06.629627 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:06.629458    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm" podUID="9b0aaae7-2a00-4487-a054-a359353bf4a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 03 22:29:07.634157 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:07.634123    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm"
Nov 03 22:29:07.634708 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:07.634633    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm"
Nov 03 22:29:07.635342 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:07.635319    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm"
Nov 03 22:29:07.639058 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:07.639024    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:29:07.794486 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:07.794441    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:29:08.954989 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:08.952627    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:29:09.412916 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:09.412817    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:29:09.730306 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:09.729367    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-50d738/distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r" podUID="e45b1572-6519-4c71-884a-115eeddd86cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Nov 03 22:29:09.842424 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:09.842388    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b7b10505-7645b7c844-knzm7"]
Nov 03 22:29:09.849236 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:09.849191    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b7b10505-7645b7c844-knzm7"
Nov 03 22:29:09.888784 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:09.857986    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6665b06d-4884-4946-947c-cbbd693d58c9-metrics-certs\") pod \"maas-api-e2e-migrate-b7b10505-7645b7c844-knzm7\" (UID: \"6665b06d-4884-4946-947c-cbbd693d58c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b7b10505-7645b7c844-knzm7"
Nov 03 22:29:09.889135 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:09.889101    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6665b06d-4884-4946-947c-cbbd693d58c9-openshift-service-ca\") pod \"maas-api-e2e-migrate-b7b10505-7645b7c844-knzm7\" (UID: \"6665b06d-4884-4946-947c-cbbd693d58c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b7b10505-7645b7c844-knzm7"
Nov 03 22:29:09.889502 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:09.889479    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djhdr\" (UniqueName: \"kubernetes.io/projected/6665b06d-4884-4946-947c-cbbd693d58c9-kube-api-access-djhdr\") pod \"maas-api-e2e-migrate-b7b10505-7645b7c844-knzm7\" (UID: \"6665b06d-4884-4946-947c-cbbd693d58c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b7b10505-7645b7c844-knzm7"
Nov 03 22:29:09.889674 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:09.889655    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6665b06d-4884-4946-947c-cbbd693d58c9-maas-api-tls\") pod \"maas-api-e2e-migrate-b7b10505-7645b7c844-knzm7\" (UID: \"6665b06d-4884-4946-947c-cbbd693d58c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b7b10505-7645b7c844-knzm7"
Nov 03 22:29:10.129373 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:10.116919    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-b7b10505-serving-cert\""
Nov 03 22:29:10.129373 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:10.118103    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:29:10.186241 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:10.182729    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djhdr\" (UniqueName: \"kubernetes.io/projected/6665b06d-4884-4946-947c-cbbd693d58c9-kube-api-access-djhdr\") pod \"maas-api-e2e-migrate-b7b10505-7645b7c844-knzm7\" (UID: \"6665b06d-4884-4946-947c-cbbd693d58c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b7b10505-7645b7c844-knzm7"
Nov 03 22:29:10.210939 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:10.210900    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6665b06d-4884-4946-947c-cbbd693d58c9-maas-api-tls\") pod \"maas-api-e2e-migrate-b7b10505-7645b7c844-knzm7\" (UID: \"6665b06d-4884-4946-947c-cbbd693d58c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b7b10505-7645b7c844-knzm7"
Nov 03 22:29:10.211405 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:10.211379    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6665b06d-4884-4946-947c-cbbd693d58c9-metrics-certs\") pod \"maas-api-e2e-migrate-b7b10505-7645b7c844-knzm7\" (UID: \"6665b06d-4884-4946-947c-cbbd693d58c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b7b10505-7645b7c844-knzm7"
Nov 03 22:29:10.220364 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:10.220329    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6665b06d-4884-4946-947c-cbbd693d58c9-openshift-service-ca\") pod \"maas-api-e2e-migrate-b7b10505-7645b7c844-knzm7\" (UID: \"6665b06d-4884-4946-947c-cbbd693d58c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b7b10505-7645b7c844-knzm7"
Nov 03 22:29:10.244238 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:10.225320    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6665b06d-4884-4946-947c-cbbd693d58c9-openshift-service-ca\") pod \"maas-api-e2e-migrate-b7b10505-7645b7c844-knzm7\" (UID: \"6665b06d-4884-4946-947c-cbbd693d58c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b7b10505-7645b7c844-knzm7"
Nov 03 22:29:10.361645 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:29:10.271453    4431 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-b7b10505-serving-cert: secret "maas-api-e2e-migrate-b7b10505-serving-cert" not found
Nov 03 22:29:10.361645 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:29:10.345351    4431 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6665b06d-4884-4946-947c-cbbd693d58c9-maas-api-tls podName:6665b06d-4884-4946-947c-cbbd693d58c9 nodeName:}" failed. No retries permitted until 2026-11-03 22:29:10.845022181 +0000 UTC m=+974.870878039 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6665b06d-4884-4946-947c-cbbd693d58c9-maas-api-tls") pod "maas-api-e2e-migrate-b7b10505-7645b7c844-knzm7" (UID: "6665b06d-4884-4946-947c-cbbd693d58c9") : secret "maas-api-e2e-migrate-b7b10505-serving-cert" not found
Nov 03 22:29:10.361645 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:10.283040    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djhdr\" (UniqueName: \"kubernetes.io/projected/6665b06d-4884-4946-947c-cbbd693d58c9-kube-api-access-djhdr\") pod \"maas-api-e2e-migrate-b7b10505-7645b7c844-knzm7\" (UID: \"6665b06d-4884-4946-947c-cbbd693d58c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b7b10505-7645b7c844-knzm7"
Nov 03 22:29:10.361645 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:10.305706    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:29:10.361645 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:10.186873    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5187e1/distinct-w3-5187e1-kserve-545b7fb6bc-mk49g" podUID="3ba8dc17-1b1b-4811-9dd6-595d75c6ca04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:10.399295 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:10.388404    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-50d738/premium-w4-50d738-kserve-59ccd7f995-xbfq2" podUID="fa0a680b-8f0d-4b19-8217-82282764511a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:10.399295 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:10.397342    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-553753/distinct-2-w2-553753-kserve-5bf8568844-7q4l2" podUID="a1894b80-164c-4ce1-94f3-1e5011d9759e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:10.403239 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:10.400159    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b7b10505-7645b7c844-knzm7"]
Nov 03 22:29:10.491241 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:10.490816    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6665b06d-4884-4946-947c-cbbd693d58c9-metrics-certs\") pod \"maas-api-e2e-migrate-b7b10505-7645b7c844-knzm7\" (UID: \"6665b06d-4884-4946-947c-cbbd693d58c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b7b10505-7645b7c844-knzm7"
Nov 03 22:29:10.588983 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:10.588314    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-50d738/distinct-w4-50d738-kserve-668fdf8b86-5pp9p" podUID="ce50af62-9cb6-4379-99c7-b40630538f4f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:10.729313 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:10.663310    4431 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5187e1/premium-w3-5187e1-kserve-56bf8788d4-pp8st" podUID="a70ecfe9-020a-4331-b67c-aaaa46ed1eb1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": context deadline exceeded"
Nov 03 22:29:10.881242 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:10.877335    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6665b06d-4884-4946-947c-cbbd693d58c9-maas-api-tls\") pod \"maas-api-e2e-migrate-b7b10505-7645b7c844-knzm7\" (UID: \"6665b06d-4884-4946-947c-cbbd693d58c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b7b10505-7645b7c844-knzm7"
Nov 03 22:29:10.906241 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:29:10.880472    4431 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-b7b10505-serving-cert: secret "maas-api-e2e-migrate-b7b10505-serving-cert" not found
Nov 03 22:29:10.906568 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:29:10.906542    4431 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6665b06d-4884-4946-947c-cbbd693d58c9-maas-api-tls podName:6665b06d-4884-4946-947c-cbbd693d58c9 nodeName:}" failed. No retries permitted until 2026-11-03 22:29:11.90650916 +0000 UTC m=+975.932365022 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6665b06d-4884-4946-947c-cbbd693d58c9-maas-api-tls") pod "maas-api-e2e-migrate-b7b10505-7645b7c844-knzm7" (UID: "6665b06d-4884-4946-947c-cbbd693d58c9") : secret "maas-api-e2e-migrate-b7b10505-serving-cert" not found
Nov 03 22:29:10.917422 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:10.917379    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5187e1/premium-w3-5187e1-kserve-56bf8788d4-pp8st" podUID="a70ecfe9-020a-4331-b67c-aaaa46ed1eb1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:11.073295 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:11.072315    4431 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-553753/simulator-w2-553753-kserve-64d77d4dfc-4rlbn" podUID="a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": context deadline exceeded"
Nov 03 22:29:11.100237 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:11.100083    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/simulator-w0-88a4d9-kserve-785f84888f-sjbzp" podUID="18bd460b-a0f1-415c-bcb4-41c9f70d2926" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:11.122803 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:11.122755    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/premium-w0-88a4d9-kserve-5559d75b7b-vqllq" podUID="05206737-4ca3-41fc-b35a-37c625cc182c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 03 22:29:11.271390 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:11.271331    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-w6-2d619f-kserve-74dbc56695-tnvtb" podUID="90310d72-c818-4fdd-b4e7-b51f633caa05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:11.349444 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:11.349338    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5187e1/distinct-w3-5187e1-kserve-545b7fb6bc-mk49g" podUID="3ba8dc17-1b1b-4811-9dd6-595d75c6ca04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:11.350567 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:11.350533    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5187e1/distinct-w3-5187e1-kserve-545b7fb6bc-mk49g"
Nov 03 22:29:11.408583 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:11.408539    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-553753/distinct-2-w2-553753-kserve-5bf8568844-7q4l2" podUID="a1894b80-164c-4ce1-94f3-1e5011d9759e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:11.409960 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:11.409933    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-553753/distinct-2-w2-553753-kserve-5bf8568844-7q4l2"
Nov 03 22:29:11.756238 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:11.753883    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-50d738/distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r" podUID="e45b1572-6519-4c71-884a-115eeddd86cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:11.914373 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:11.914320    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6665b06d-4884-4946-947c-cbbd693d58c9-maas-api-tls\") pod \"maas-api-e2e-migrate-b7b10505-7645b7c844-knzm7\" (UID: \"6665b06d-4884-4946-947c-cbbd693d58c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b7b10505-7645b7c844-knzm7"
Nov 03 22:29:11.914894 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:29:11.914708    4431 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-b7b10505-serving-cert: secret "maas-api-e2e-migrate-b7b10505-serving-cert" not found
Nov 03 22:29:11.915079 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:29:11.915069    4431 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6665b06d-4884-4946-947c-cbbd693d58c9-maas-api-tls podName:6665b06d-4884-4946-947c-cbbd693d58c9 nodeName:}" failed. No retries permitted until 2026-11-03 22:29:13.915047201 +0000 UTC m=+977.940903066 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6665b06d-4884-4946-947c-cbbd693d58c9-maas-api-tls") pod "maas-api-e2e-migrate-b7b10505-7645b7c844-knzm7" (UID: "6665b06d-4884-4946-947c-cbbd693d58c9") : secret "maas-api-e2e-migrate-b7b10505-serving-cert" not found
Nov 03 22:29:12.339734 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:12.339688    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-553753/simulator-w2-553753-kserve-64d77d4dfc-4rlbn" podUID="a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:12.340864 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:12.340347    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-50d738/premium-w4-50d738-kserve-59ccd7f995-xbfq2" podUID="fa0a680b-8f0d-4b19-8217-82282764511a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:12.350329 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:12.350293    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5187e1/distinct-w3-5187e1-kserve-545b7fb6bc-mk49g" podUID="3ba8dc17-1b1b-4811-9dd6-595d75c6ca04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Nov 03 22:29:12.410003 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:12.409848    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-553753/distinct-2-w2-553753-kserve-5bf8568844-7q4l2" podUID="a1894b80-164c-4ce1-94f3-1e5011d9759e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:12.506245 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:12.505334    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-50d738/distinct-w4-50d738-kserve-668fdf8b86-5pp9p" podUID="ce50af62-9cb6-4379-99c7-b40630538f4f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:12.520270 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:12.518341    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5187e1/premium-w3-5187e1-kserve-56bf8788d4-pp8st" podUID="a70ecfe9-020a-4331-b67c-aaaa46ed1eb1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:12.669120 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:12.669088    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-553753/distinct-2-w2-553753-kserve-5bf8568844-7q4l2"
Nov 03 22:29:12.760394 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:12.760344    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-50d738/distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r" podUID="e45b1572-6519-4c71-884a-115eeddd86cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Nov 03 22:29:12.761494 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:12.761470    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-50d738/distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r"
Nov 03 22:29:12.785959 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:12.785823    4431 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5187e1/distinct-w3-5187e1-kserve-545b7fb6bc-mk49g" podUID="3ba8dc17-1b1b-4811-9dd6-595d75c6ca04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:12.997327 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:12.983780    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/simulator-w0-88a4d9-kserve-785f84888f-sjbzp" podUID="18bd460b-a0f1-415c-bcb4-41c9f70d2926" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:13.174371 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:13.174324    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/premium-w0-88a4d9-kserve-5559d75b7b-vqllq" podUID="05206737-4ca3-41fc-b35a-37c625cc182c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 03 22:29:13.175011 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:13.174978    4431 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-50d738/premium-w4-50d738-kserve-59ccd7f995-xbfq2" podUID="fa0a680b-8f0d-4b19-8217-82282764511a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": context deadline exceeded"
Nov 03 22:29:13.361847 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:13.361742    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5187e1/distinct-w3-5187e1-kserve-545b7fb6bc-mk49g" podUID="3ba8dc17-1b1b-4811-9dd6-595d75c6ca04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:13.364959 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:13.364922    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-553753/simulator-w2-553753-kserve-64d77d4dfc-4rlbn" podUID="a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:13.385507 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:13.385466    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-553753/simulator-w2-553753-kserve-64d77d4dfc-4rlbn"
Nov 03 22:29:13.546241 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:13.511120    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-50d738/distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r"
Nov 03 22:29:13.968363 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:13.590348    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-50d738/distinct-w4-50d738-kserve-668fdf8b86-5pp9p" podUID="ce50af62-9cb6-4379-99c7-b40630538f4f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:13.968993 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:13.637836    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:29:13.994242 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:13.969700    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-50d738/distinct-w4-50d738-kserve-668fdf8b86-5pp9p"
Nov 03 22:29:13.994557 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:13.707066    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5187e1/premium-w3-5187e1-kserve-56bf8788d4-pp8st" podUID="a70ecfe9-020a-4331-b67c-aaaa46ed1eb1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:14.032479 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:14.032437    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/simulator-w0-88a4d9-kserve-785f84888f-sjbzp" podUID="18bd460b-a0f1-415c-bcb4-41c9f70d2926" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded"
Nov 03 22:29:14.035044 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:14.035021    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5187e1/premium-w3-5187e1-kserve-56bf8788d4-pp8st"
Nov 03 22:29:14.035472 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:14.035446    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-88a4d9/simulator-w0-88a4d9-kserve-785f84888f-sjbzp"
Nov 03 22:29:14.265418 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:14.265319    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/premium-w0-88a4d9-kserve-5559d75b7b-vqllq" podUID="05206737-4ca3-41fc-b35a-37c625cc182c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:14.275583 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:14.275527    4431 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-88a4d9/premium-w0-88a4d9-kserve-5559d75b7b-vqllq" podUID="05206737-4ca3-41fc-b35a-37c625cc182c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:14.305240 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:14.303273    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6665b06d-4884-4946-947c-cbbd693d58c9-maas-api-tls\") pod \"maas-api-e2e-migrate-b7b10505-7645b7c844-knzm7\" (UID: \"6665b06d-4884-4946-947c-cbbd693d58c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b7b10505-7645b7c844-knzm7"
Nov 03 22:29:14.323016 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:14.322982    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:29:14.325787 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:14.325753    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-88a4d9/premium-w0-88a4d9-kserve-5559d75b7b-vqllq"
Nov 03 22:29:14.325948 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:14.325820    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-88a4d9/premium-w0-88a4d9-kserve-5559d75b7b-vqllq"
Nov 03 22:29:14.325948 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:14.325854    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5187e1/distinct-w3-5187e1-kserve-545b7fb6bc-mk49g"
Nov 03 22:29:14.330448 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:14.330352    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6665b06d-4884-4946-947c-cbbd693d58c9-maas-api-tls\") pod \"maas-api-e2e-migrate-b7b10505-7645b7c844-knzm7\" (UID: \"6665b06d-4884-4946-947c-cbbd693d58c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b7b10505-7645b7c844-knzm7"
Nov 03 22:29:14.364161 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:14.364132    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-88a4d9/simulator-w0-88a4d9-kserve-785f84888f-sjbzp"
Nov 03 22:29:14.365327 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:14.365280    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-553753/simulator-w2-553753-kserve-64d77d4dfc-4rlbn" podUID="a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:14.366923 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:14.366907    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b7b10505-7645b7c844-knzm7"
Nov 03 22:29:14.397914 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:14.396630    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5187e1/premium-w3-5187e1-kserve-56bf8788d4-pp8st"
Nov 03 22:29:14.624439 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:14.624289    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-553753/simulator-w2-553753-kserve-64d77d4dfc-4rlbn"
Nov 03 22:29:14.627670 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:14.627237    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-50d738/distinct-w4-50d738-kserve-668fdf8b86-5pp9p"
Nov 03 22:29:14.973961 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:14.973928    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b7b10505-7645b7c844-knzm7"]
Nov 03 22:29:14.975749 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:29:14.975721    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6665b06d_4884_4946_947c_cbbd693d58c9.slice/crio-03551a9c6f0d9c29a83d492fc229f34802fd57c1d05b0c5cd1d6271238ba66c6 WatchSource:0}: Error finding container 03551a9c6f0d9c29a83d492fc229f34802fd57c1d05b0c5cd1d6271238ba66c6: Status 404 returned error can't find the container with id 03551a9c6f0d9c29a83d492fc229f34802fd57c1d05b0c5cd1d6271238ba66c6
Nov 03 22:29:14.977195 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:14.977175    4431 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 22:29:15.274950 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:15.274854    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b7b10505-7645b7c844-knzm7" event={"ID":"6665b06d-4884-4946-947c-cbbd693d58c9","Type":"ContainerStarted","Data":"03551a9c6f0d9c29a83d492fc229f34802fd57c1d05b0c5cd1d6271238ba66c6"}
Nov 03 22:29:15.609676 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:15.609584    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:29:16.282875 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:16.282844    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b7b10505-7645b7c844-knzm7" event={"ID":"6665b06d-4884-4946-947c-cbbd693d58c9","Type":"ContainerStarted","Data":"8e7e11d6419b51f321e3da9ef2ae85a75d5d7956f1e9af9dd20c07d228744a1c"}
Nov 03 22:29:16.283058 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:16.282899    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b7b10505-7645b7c844-knzm7"
Nov 03 22:29:16.327337 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:16.327277    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b7b10505-7645b7c844-knzm7" podStartSLOduration=6.888115472 podStartE2EDuration="7.32725822s" totalImagesPullingTime="439.142748ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:29:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:29:16.3249365 +0000 UTC m=+980.350792363" watchObservedRunningTime="2026-11-03 22:29:16.32725822 +0000 UTC m=+980.353114086"
Nov 03 22:29:16.445605 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:16.445570    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:29:17.272762 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:17.272721    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:29:17.429167 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:17.429125    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:29:18.803887 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:18.803852    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:29:19.115347 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:19.115249    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:29:20.422958 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:20.422928    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:29:21.793175 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:21.793130    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:29:21.959150 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:21.959120    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:29:22.214759 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:22.214726    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:29:22.295114 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:22.295085    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b7b10505-7645b7c844-knzm7"
Nov 03 22:29:24.023959 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:24.023922    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:29:24.496577 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:24.496544    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:29:24.654632 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:24.654578    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:29:24.738329 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:24.738292    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-51bfa2e1-6954676576-kr8sq"]
Nov 03 22:29:24.742550 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:24.742530    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-51bfa2e1-6954676576-kr8sq"
Nov 03 22:29:24.761050 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:24.761019    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-51bfa2e1-6954676576-kr8sq"]
Nov 03 22:29:24.768758 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:24.768542    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-51bfa2e1\""
Nov 03 22:29:24.809541 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:24.809504    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:29:24.814123 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:24.814095    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e679b58-cace-47c0-8673-3f9e9addea36-plugins-config-volume\") pod \"payload-processing-e2e-ait-51bfa2e1-6954676576-kr8sq\" (UID: \"5e679b58-cace-47c0-8673-3f9e9addea36\") " pod="openshift-ingress/payload-processing-e2e-ait-51bfa2e1-6954676576-kr8sq"
Nov 03 22:29:24.814261 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:24.814147    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tdfp\" (UniqueName: \"kubernetes.io/projected/5e679b58-cace-47c0-8673-3f9e9addea36-kube-api-access-9tdfp\") pod \"payload-processing-e2e-ait-51bfa2e1-6954676576-kr8sq\" (UID: \"5e679b58-cace-47c0-8673-3f9e9addea36\") " pod="openshift-ingress/payload-processing-e2e-ait-51bfa2e1-6954676576-kr8sq"
Nov 03 22:29:24.931753 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:24.931719    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e679b58-cace-47c0-8673-3f9e9addea36-plugins-config-volume\") pod \"payload-processing-e2e-ait-51bfa2e1-6954676576-kr8sq\" (UID: \"5e679b58-cace-47c0-8673-3f9e9addea36\") " pod="openshift-ingress/payload-processing-e2e-ait-51bfa2e1-6954676576-kr8sq"
Nov 03 22:29:24.932066 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:24.932034    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tdfp\" (UniqueName: \"kubernetes.io/projected/5e679b58-cace-47c0-8673-3f9e9addea36-kube-api-access-9tdfp\") pod \"payload-processing-e2e-ait-51bfa2e1-6954676576-kr8sq\" (UID: \"5e679b58-cace-47c0-8673-3f9e9addea36\") " pod="openshift-ingress/payload-processing-e2e-ait-51bfa2e1-6954676576-kr8sq"
Nov 03 22:29:24.933584 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:24.933559    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e679b58-cace-47c0-8673-3f9e9addea36-plugins-config-volume\") pod \"payload-processing-e2e-ait-51bfa2e1-6954676576-kr8sq\" (UID: \"5e679b58-cace-47c0-8673-3f9e9addea36\") " pod="openshift-ingress/payload-processing-e2e-ait-51bfa2e1-6954676576-kr8sq"
Nov 03 22:29:24.982310 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:24.973944    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tdfp\" (UniqueName: \"kubernetes.io/projected/5e679b58-cace-47c0-8673-3f9e9addea36-kube-api-access-9tdfp\") pod \"payload-processing-e2e-ait-51bfa2e1-6954676576-kr8sq\" (UID: \"5e679b58-cace-47c0-8673-3f9e9addea36\") " pod="openshift-ingress/payload-processing-e2e-ait-51bfa2e1-6954676576-kr8sq"
Nov 03 22:29:25.057532 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:25.057439    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-51bfa2e1-6954676576-kr8sq"
Nov 03 22:29:25.370520 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:25.370440    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-51bfa2e1-6954676576-kr8sq"]
Nov 03 22:29:25.375863 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:29:25.375834    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e679b58_cace_47c0_8673_3f9e9addea36.slice/crio-b35f342c4be0bb6b4c96c551fe413adeb29583707532b1d8d2372ecb6e514d66 WatchSource:0}: Error finding container b35f342c4be0bb6b4c96c551fe413adeb29583707532b1d8d2372ecb6e514d66: Status 404 returned error can't find the container with id b35f342c4be0bb6b4c96c551fe413adeb29583707532b1d8d2372ecb6e514d66
Nov 03 22:29:25.713689 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:25.713636    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:29:26.346541 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:26.346498    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-51bfa2e1-6954676576-kr8sq" event={"ID":"5e679b58-cace-47c0-8673-3f9e9addea36","Type":"ContainerStarted","Data":"b35f342c4be0bb6b4c96c551fe413adeb29583707532b1d8d2372ecb6e514d66"}
Nov 03 22:29:27.284877 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:27.284805    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:29:28.361336 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:28.361297    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-51bfa2e1-6954676576-kr8sq" event={"ID":"5e679b58-cace-47c0-8673-3f9e9addea36","Type":"ContainerStarted","Data":"0a09c5fb968f0b598019d4116cc86794adecae0eb5dfb9b4ee276620269c2bdc"}
Nov 03 22:29:28.361846 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:28.361356    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-51bfa2e1-6954676576-kr8sq"
Nov 03 22:29:28.403737 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:28.403687    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-51bfa2e1-6954676576-kr8sq" podStartSLOduration=2.293796909 podStartE2EDuration="4.403673289s" totalImagesPullingTime="2.10987638s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:29:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:29:28.400619842 +0000 UTC m=+992.426475704" watchObservedRunningTime="2026-11-03 22:29:28.403673289 +0000 UTC m=+992.429529153"
Nov 03 22:29:28.747656 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:28.747611    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr"]
Nov 03 22:29:28.751973 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:28.751950    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr"
Nov 03 22:29:28.774984 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:28.774957    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr"]
Nov 03 22:29:28.886644 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:28.886608    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:29:28.892818 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:28.892791    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffc24874-f9cc-4433-a4f4-2a72788b1067-istio-data\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr\" (UID: \"ffc24874-f9cc-4433-a4f4-2a72788b1067\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr"
Nov 03 22:29:28.892967 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:28.892853    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffc24874-f9cc-4433-a4f4-2a72788b1067-istiod-ca-cert\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr\" (UID: \"ffc24874-f9cc-4433-a4f4-2a72788b1067\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr"
Nov 03 22:29:28.892967 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:28.892915    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdc4r\" (UniqueName: \"kubernetes.io/projected/ffc24874-f9cc-4433-a4f4-2a72788b1067-kube-api-access-fdc4r\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr\" (UID: \"ffc24874-f9cc-4433-a4f4-2a72788b1067\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr"
Nov 03 22:29:28.893044 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:28.892968    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffc24874-f9cc-4433-a4f4-2a72788b1067-credential-socket\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr\" (UID: \"ffc24874-f9cc-4433-a4f4-2a72788b1067\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr"
Nov 03 22:29:28.893044 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:28.892993    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffc24874-f9cc-4433-a4f4-2a72788b1067-istio-envoy\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr\" (UID: \"ffc24874-f9cc-4433-a4f4-2a72788b1067\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr"
Nov 03 22:29:28.893152 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:28.893129    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffc24874-f9cc-4433-a4f4-2a72788b1067-istio-podinfo\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr\" (UID: \"ffc24874-f9cc-4433-a4f4-2a72788b1067\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr"
Nov 03 22:29:28.893201 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:28.893185    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffc24874-f9cc-4433-a4f4-2a72788b1067-workload-socket\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr\" (UID: \"ffc24874-f9cc-4433-a4f4-2a72788b1067\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr"
Nov 03 22:29:28.893259 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:28.893235    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffc24874-f9cc-4433-a4f4-2a72788b1067-istio-token\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr\" (UID: \"ffc24874-f9cc-4433-a4f4-2a72788b1067\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr"
Nov 03 22:29:28.893339 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:28.893323    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffc24874-f9cc-4433-a4f4-2a72788b1067-workload-certs\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr\" (UID: \"ffc24874-f9cc-4433-a4f4-2a72788b1067\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr"
Nov 03 22:29:28.935338 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:28.935303    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc"]
Nov 03 22:29:28.939706 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:28.939680    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc"
Nov 03 22:29:28.963505 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:28.963466    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc"]
Nov 03 22:29:28.970111 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:28.970082    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t"]
Nov 03 22:29:28.974673 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:28.974645    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t"
Nov 03 22:29:28.995468 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:28.995430    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffc24874-f9cc-4433-a4f4-2a72788b1067-istio-data\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr\" (UID: \"ffc24874-f9cc-4433-a4f4-2a72788b1067\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr"
Nov 03 22:29:28.996107 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:28.995527    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffc24874-f9cc-4433-a4f4-2a72788b1067-istiod-ca-cert\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr\" (UID: \"ffc24874-f9cc-4433-a4f4-2a72788b1067\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr"
Nov 03 22:29:28.996107 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:28.995558    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdc4r\" (UniqueName: \"kubernetes.io/projected/ffc24874-f9cc-4433-a4f4-2a72788b1067-kube-api-access-fdc4r\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr\" (UID: \"ffc24874-f9cc-4433-a4f4-2a72788b1067\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr"
Nov 03 22:29:28.996107 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:28.995635    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffc24874-f9cc-4433-a4f4-2a72788b1067-credential-socket\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr\" (UID: \"ffc24874-f9cc-4433-a4f4-2a72788b1067\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr"
Nov 03 22:29:28.996107 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:28.995665    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffc24874-f9cc-4433-a4f4-2a72788b1067-istio-envoy\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr\" (UID: \"ffc24874-f9cc-4433-a4f4-2a72788b1067\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr"
Nov 03 22:29:28.996107 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:28.995692    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffc24874-f9cc-4433-a4f4-2a72788b1067-istio-podinfo\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr\" (UID: \"ffc24874-f9cc-4433-a4f4-2a72788b1067\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr"
Nov 03 22:29:28.996107 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:28.995747    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffc24874-f9cc-4433-a4f4-2a72788b1067-workload-socket\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr\" (UID: \"ffc24874-f9cc-4433-a4f4-2a72788b1067\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr"
Nov 03 22:29:28.996107 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:28.995771    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffc24874-f9cc-4433-a4f4-2a72788b1067-istio-token\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr\" (UID: \"ffc24874-f9cc-4433-a4f4-2a72788b1067\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr"
Nov 03 22:29:28.996107 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:28.995872    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffc24874-f9cc-4433-a4f4-2a72788b1067-workload-certs\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr\" (UID: \"ffc24874-f9cc-4433-a4f4-2a72788b1067\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr"
Nov 03 22:29:28.997177 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:28.996453    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffc24874-f9cc-4433-a4f4-2a72788b1067-workload-certs\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr\" (UID: \"ffc24874-f9cc-4433-a4f4-2a72788b1067\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr"
Nov 03 22:29:28.997177 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:28.996759    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffc24874-f9cc-4433-a4f4-2a72788b1067-istio-data\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr\" (UID: \"ffc24874-f9cc-4433-a4f4-2a72788b1067\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr"
Nov 03 22:29:28.997769 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:28.997405    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffc24874-f9cc-4433-a4f4-2a72788b1067-istiod-ca-cert\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr\" (UID: \"ffc24874-f9cc-4433-a4f4-2a72788b1067\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr"
Nov 03 22:29:28.997999 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:28.997967    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffc24874-f9cc-4433-a4f4-2a72788b1067-credential-socket\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr\" (UID: \"ffc24874-f9cc-4433-a4f4-2a72788b1067\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr"
Nov 03 22:29:29.009166 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:28.998805    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffc24874-f9cc-4433-a4f4-2a72788b1067-workload-socket\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr\" (UID: \"ffc24874-f9cc-4433-a4f4-2a72788b1067\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr"
Nov 03 22:29:29.009166 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.004144    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t"]
Nov 03 22:29:29.009166 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.006504    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffc24874-f9cc-4433-a4f4-2a72788b1067-istio-envoy\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr\" (UID: \"ffc24874-f9cc-4433-a4f4-2a72788b1067\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr"
Nov 03 22:29:29.009941 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.009194    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2"]
Nov 03 22:29:29.012026 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.012002    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffc24874-f9cc-4433-a4f4-2a72788b1067-istio-podinfo\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr\" (UID: \"ffc24874-f9cc-4433-a4f4-2a72788b1067\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr"
Nov 03 22:29:29.014885 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.014862    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2"
Nov 03 22:29:29.030645 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.030611    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2"]
Nov 03 22:29:29.033922 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.033892    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdc4r\" (UniqueName: \"kubernetes.io/projected/ffc24874-f9cc-4433-a4f4-2a72788b1067-kube-api-access-fdc4r\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr\" (UID: \"ffc24874-f9cc-4433-a4f4-2a72788b1067\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr"
Nov 03 22:29:29.035527 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.035504    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffc24874-f9cc-4433-a4f4-2a72788b1067-istio-token\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr\" (UID: \"ffc24874-f9cc-4433-a4f4-2a72788b1067\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr"
Nov 03 22:29:29.066901 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.066867    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr"
Nov 03 22:29:29.104264 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.098278    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36b9d353-1537-445b-bb03-d2d917a2a208-workload-certs\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc\" (UID: \"36b9d353-1537-445b-bb03-d2d917a2a208\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc"
Nov 03 22:29:29.104264 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.098438    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3d85fcc-b84d-48a3-944a-810ce94062a2-workload-certs\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t\" (UID: \"e3d85fcc-b84d-48a3-944a-810ce94062a2\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t"
Nov 03 22:29:29.104264 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.098588    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36b9d353-1537-445b-bb03-d2d917a2a208-istio-token\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc\" (UID: \"36b9d353-1537-445b-bb03-d2d917a2a208\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc"
Nov 03 22:29:29.104264 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.098648    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3d85fcc-b84d-48a3-944a-810ce94062a2-workload-socket\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t\" (UID: \"e3d85fcc-b84d-48a3-944a-810ce94062a2\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t"
Nov 03 22:29:29.104264 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.098749    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k28v6\" (UniqueName: \"kubernetes.io/projected/e3d85fcc-b84d-48a3-944a-810ce94062a2-kube-api-access-k28v6\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t\" (UID: \"e3d85fcc-b84d-48a3-944a-810ce94062a2\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t"
Nov 03 22:29:29.104264 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.098983    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zns5l\" (UniqueName: \"kubernetes.io/projected/36b9d353-1537-445b-bb03-d2d917a2a208-kube-api-access-zns5l\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc\" (UID: \"36b9d353-1537-445b-bb03-d2d917a2a208\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc"
Nov 03 22:29:29.104264 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.099032    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36b9d353-1537-445b-bb03-d2d917a2a208-credential-socket\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc\" (UID: \"36b9d353-1537-445b-bb03-d2d917a2a208\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc"
Nov 03 22:29:29.104264 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.099118    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36b9d353-1537-445b-bb03-d2d917a2a208-istiod-ca-cert\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc\" (UID: \"36b9d353-1537-445b-bb03-d2d917a2a208\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc"
Nov 03 22:29:29.104264 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.099175    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36b9d353-1537-445b-bb03-d2d917a2a208-istio-data\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc\" (UID: \"36b9d353-1537-445b-bb03-d2d917a2a208\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc"
Nov 03 22:29:29.104264 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.099208    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3d85fcc-b84d-48a3-944a-810ce94062a2-istio-podinfo\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t\" (UID: \"e3d85fcc-b84d-48a3-944a-810ce94062a2\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t"
Nov 03 22:29:29.104264 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.099296    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3d85fcc-b84d-48a3-944a-810ce94062a2-istio-data\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t\" (UID: \"e3d85fcc-b84d-48a3-944a-810ce94062a2\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t"
Nov 03 22:29:29.104264 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.099326    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36b9d353-1537-445b-bb03-d2d917a2a208-istio-envoy\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc\" (UID: \"36b9d353-1537-445b-bb03-d2d917a2a208\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc"
Nov 03 22:29:29.104264 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.099382    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3d85fcc-b84d-48a3-944a-810ce94062a2-credential-socket\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t\" (UID: \"e3d85fcc-b84d-48a3-944a-810ce94062a2\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t"
Nov 03 22:29:29.104264 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.099411    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36b9d353-1537-445b-bb03-d2d917a2a208-istio-podinfo\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc\" (UID: \"36b9d353-1537-445b-bb03-d2d917a2a208\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc"
Nov 03 22:29:29.104264 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.099444    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3d85fcc-b84d-48a3-944a-810ce94062a2-istiod-ca-cert\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t\" (UID: \"e3d85fcc-b84d-48a3-944a-810ce94062a2\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t"
Nov 03 22:29:29.104264 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.099473    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3d85fcc-b84d-48a3-944a-810ce94062a2-istio-envoy\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t\" (UID: \"e3d85fcc-b84d-48a3-944a-810ce94062a2\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t"
Nov 03 22:29:29.104264 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.099507    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3d85fcc-b84d-48a3-944a-810ce94062a2-istio-token\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t\" (UID: \"e3d85fcc-b84d-48a3-944a-810ce94062a2\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t"
Nov 03 22:29:29.104264 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.099599    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36b9d353-1537-445b-bb03-d2d917a2a208-workload-socket\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc\" (UID: \"36b9d353-1537-445b-bb03-d2d917a2a208\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc"
Nov 03 22:29:29.200700 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.200658    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36b9d353-1537-445b-bb03-d2d917a2a208-istio-envoy\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc\" (UID: \"36b9d353-1537-445b-bb03-d2d917a2a208\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc"
Nov 03 22:29:29.200891 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.200750    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3d85fcc-b84d-48a3-944a-810ce94062a2-credential-socket\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t\" (UID: \"e3d85fcc-b84d-48a3-944a-810ce94062a2\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t"
Nov 03 22:29:29.200891 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.200782    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36b9d353-1537-445b-bb03-d2d917a2a208-istio-podinfo\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc\" (UID: \"36b9d353-1537-445b-bb03-d2d917a2a208\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc"
Nov 03 22:29:29.200891 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.200817    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3d85fcc-b84d-48a3-944a-810ce94062a2-istiod-ca-cert\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t\" (UID: \"e3d85fcc-b84d-48a3-944a-810ce94062a2\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t"
Nov 03 22:29:29.200891 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.200847    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3d85fcc-b84d-48a3-944a-810ce94062a2-istio-envoy\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t\" (UID: \"e3d85fcc-b84d-48a3-944a-810ce94062a2\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t"
Nov 03 22:29:29.200891 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.200880    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3d85fcc-b84d-48a3-944a-810ce94062a2-istio-token\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t\" (UID: \"e3d85fcc-b84d-48a3-944a-810ce94062a2\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t"
Nov 03 22:29:29.201161 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.200962    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ace43d0-fe92-4e5e-a190-26408d4ab47b-istio-data\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2\" (UID: \"8ace43d0-fe92-4e5e-a190-26408d4ab47b\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2"
Nov 03 22:29:29.201161 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.201003    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36b9d353-1537-445b-bb03-d2d917a2a208-workload-socket\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc\" (UID: \"36b9d353-1537-445b-bb03-d2d917a2a208\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc"
Nov 03 22:29:29.201161 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.201030    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36b9d353-1537-445b-bb03-d2d917a2a208-workload-certs\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc\" (UID: \"36b9d353-1537-445b-bb03-d2d917a2a208\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc"
Nov 03 22:29:29.201161 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.201058    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3d85fcc-b84d-48a3-944a-810ce94062a2-workload-certs\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t\" (UID: \"e3d85fcc-b84d-48a3-944a-810ce94062a2\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t"
Nov 03 22:29:29.201161 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.201087    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ace43d0-fe92-4e5e-a190-26408d4ab47b-istio-envoy\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2\" (UID: \"8ace43d0-fe92-4e5e-a190-26408d4ab47b\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2"
Nov 03 22:29:29.201161 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.201144    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36b9d353-1537-445b-bb03-d2d917a2a208-istio-token\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc\" (UID: \"36b9d353-1537-445b-bb03-d2d917a2a208\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc"
Nov 03 22:29:29.201587 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.201202    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3d85fcc-b84d-48a3-944a-810ce94062a2-credential-socket\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t\" (UID: \"e3d85fcc-b84d-48a3-944a-810ce94062a2\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t"
Nov 03 22:29:29.201717 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.201693    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36b9d353-1537-445b-bb03-d2d917a2a208-workload-certs\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc\" (UID: \"36b9d353-1537-445b-bb03-d2d917a2a208\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc"
Nov 03 22:29:29.202396 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.201931    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3d85fcc-b84d-48a3-944a-810ce94062a2-istiod-ca-cert\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t\" (UID: \"e3d85fcc-b84d-48a3-944a-810ce94062a2\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t"
Nov 03 22:29:29.202396 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.202243    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36b9d353-1537-445b-bb03-d2d917a2a208-workload-socket\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc\" (UID: \"36b9d353-1537-445b-bb03-d2d917a2a208\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc"
Nov 03 22:29:29.202396 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.202324    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ace43d0-fe92-4e5e-a190-26408d4ab47b-workload-socket\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2\" (UID: \"8ace43d0-fe92-4e5e-a190-26408d4ab47b\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2"
Nov 03 22:29:29.202396 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.202367    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3d85fcc-b84d-48a3-944a-810ce94062a2-workload-socket\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t\" (UID: \"e3d85fcc-b84d-48a3-944a-810ce94062a2\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t"
Nov 03 22:29:29.202668 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.202417    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ace43d0-fe92-4e5e-a190-26408d4ab47b-workload-certs\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2\" (UID: \"8ace43d0-fe92-4e5e-a190-26408d4ab47b\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2"
Nov 03 22:29:29.202668 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.202446    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3d85fcc-b84d-48a3-944a-810ce94062a2-workload-certs\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t\" (UID: \"e3d85fcc-b84d-48a3-944a-810ce94062a2\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t"
Nov 03 22:29:29.202668 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.202511    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67fzs\" (UniqueName: \"kubernetes.io/projected/8ace43d0-fe92-4e5e-a190-26408d4ab47b-kube-api-access-67fzs\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2\" (UID: \"8ace43d0-fe92-4e5e-a190-26408d4ab47b\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2"
Nov 03 22:29:29.202668 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.202550    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k28v6\" (UniqueName: \"kubernetes.io/projected/e3d85fcc-b84d-48a3-944a-810ce94062a2-kube-api-access-k28v6\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t\" (UID: \"e3d85fcc-b84d-48a3-944a-810ce94062a2\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t"
Nov 03 22:29:29.202668 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.202588    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ace43d0-fe92-4e5e-a190-26408d4ab47b-istio-token\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2\" (UID: \"8ace43d0-fe92-4e5e-a190-26408d4ab47b\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2"
Nov 03 22:29:29.202928 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.202683    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3d85fcc-b84d-48a3-944a-810ce94062a2-workload-socket\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t\" (UID: \"e3d85fcc-b84d-48a3-944a-810ce94062a2\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t"
Nov 03 22:29:29.202928 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.202690    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ace43d0-fe92-4e5e-a190-26408d4ab47b-credential-socket\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2\" (UID: \"8ace43d0-fe92-4e5e-a190-26408d4ab47b\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2"
Nov 03 22:29:29.202928 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.202911    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zns5l\" (UniqueName: \"kubernetes.io/projected/36b9d353-1537-445b-bb03-d2d917a2a208-kube-api-access-zns5l\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc\" (UID: \"36b9d353-1537-445b-bb03-d2d917a2a208\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc"
Nov 03 22:29:29.203083 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.202967    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36b9d353-1537-445b-bb03-d2d917a2a208-credential-socket\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc\" (UID: \"36b9d353-1537-445b-bb03-d2d917a2a208\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc"
Nov 03 22:29:29.203083 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.203016    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36b9d353-1537-445b-bb03-d2d917a2a208-istiod-ca-cert\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc\" (UID: \"36b9d353-1537-445b-bb03-d2d917a2a208\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc"
Nov 03 22:29:29.203083 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.203073    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36b9d353-1537-445b-bb03-d2d917a2a208-istio-data\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc\" (UID: \"36b9d353-1537-445b-bb03-d2d917a2a208\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc"
Nov 03 22:29:29.203271 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.203103    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ace43d0-fe92-4e5e-a190-26408d4ab47b-istio-podinfo\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2\" (UID: \"8ace43d0-fe92-4e5e-a190-26408d4ab47b\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2"
Nov 03 22:29:29.203271 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.203134    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3d85fcc-b84d-48a3-944a-810ce94062a2-istio-podinfo\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t\" (UID: \"e3d85fcc-b84d-48a3-944a-810ce94062a2\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t"
Nov 03 22:29:29.203271 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.203181    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ace43d0-fe92-4e5e-a190-26408d4ab47b-istiod-ca-cert\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2\" (UID: \"8ace43d0-fe92-4e5e-a190-26408d4ab47b\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2"
Nov 03 22:29:29.203271 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.203259    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3d85fcc-b84d-48a3-944a-810ce94062a2-istio-data\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t\" (UID: \"e3d85fcc-b84d-48a3-944a-810ce94062a2\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t"
Nov 03 22:29:29.203574 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.203535    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36b9d353-1537-445b-bb03-d2d917a2a208-istio-data\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc\" (UID: \"36b9d353-1537-445b-bb03-d2d917a2a208\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc"
Nov 03 22:29:29.203654 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.203618    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3d85fcc-b84d-48a3-944a-810ce94062a2-istio-data\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t\" (UID: \"e3d85fcc-b84d-48a3-944a-810ce94062a2\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t"
Nov 03 22:29:29.203867 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.203830    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36b9d353-1537-445b-bb03-d2d917a2a208-credential-socket\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc\" (UID: \"36b9d353-1537-445b-bb03-d2d917a2a208\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc"
Nov 03 22:29:29.204259 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.204199    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3d85fcc-b84d-48a3-944a-810ce94062a2-istio-envoy\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t\" (UID: \"e3d85fcc-b84d-48a3-944a-810ce94062a2\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t"
Nov 03 22:29:29.204594 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.204524    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36b9d353-1537-445b-bb03-d2d917a2a208-istiod-ca-cert\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc\" (UID: \"36b9d353-1537-445b-bb03-d2d917a2a208\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc"
Nov 03 22:29:29.204594 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.204555    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36b9d353-1537-445b-bb03-d2d917a2a208-istio-envoy\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc\" (UID: \"36b9d353-1537-445b-bb03-d2d917a2a208\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc"
Nov 03 22:29:29.205943 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.205917    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36b9d353-1537-445b-bb03-d2d917a2a208-istio-podinfo\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc\" (UID: \"36b9d353-1537-445b-bb03-d2d917a2a208\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc"
Nov 03 22:29:29.206473 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.206448    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3d85fcc-b84d-48a3-944a-810ce94062a2-istio-podinfo\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t\" (UID: \"e3d85fcc-b84d-48a3-944a-810ce94062a2\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t"
Nov 03 22:29:29.224519 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.224486    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36b9d353-1537-445b-bb03-d2d917a2a208-istio-token\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc\" (UID: \"36b9d353-1537-445b-bb03-d2d917a2a208\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc"
Nov 03 22:29:29.224734 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.224716    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3d85fcc-b84d-48a3-944a-810ce94062a2-istio-token\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t\" (UID: \"e3d85fcc-b84d-48a3-944a-810ce94062a2\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t"
Nov 03 22:29:29.225078 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.225054    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k28v6\" (UniqueName: \"kubernetes.io/projected/e3d85fcc-b84d-48a3-944a-810ce94062a2-kube-api-access-k28v6\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t\" (UID: \"e3d85fcc-b84d-48a3-944a-810ce94062a2\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t"
Nov 03 22:29:29.225606 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.225588    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zns5l\" (UniqueName: \"kubernetes.io/projected/36b9d353-1537-445b-bb03-d2d917a2a208-kube-api-access-zns5l\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc\" (UID: \"36b9d353-1537-445b-bb03-d2d917a2a208\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc"
Nov 03 22:29:29.255602 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.255524    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc"
Nov 03 22:29:29.308783 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.308746    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ace43d0-fe92-4e5e-a190-26408d4ab47b-istio-podinfo\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2\" (UID: \"8ace43d0-fe92-4e5e-a190-26408d4ab47b\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2"
Nov 03 22:29:29.308951 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.308814    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ace43d0-fe92-4e5e-a190-26408d4ab47b-istiod-ca-cert\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2\" (UID: \"8ace43d0-fe92-4e5e-a190-26408d4ab47b\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2"
Nov 03 22:29:29.308951 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.308945    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ace43d0-fe92-4e5e-a190-26408d4ab47b-istio-data\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2\" (UID: \"8ace43d0-fe92-4e5e-a190-26408d4ab47b\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2"
Nov 03 22:29:29.309043 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.308989    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ace43d0-fe92-4e5e-a190-26408d4ab47b-istio-envoy\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2\" (UID: \"8ace43d0-fe92-4e5e-a190-26408d4ab47b\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2"
Nov 03 22:29:29.309096 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.309043    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ace43d0-fe92-4e5e-a190-26408d4ab47b-workload-socket\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2\" (UID: \"8ace43d0-fe92-4e5e-a190-26408d4ab47b\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2"
Nov 03 22:29:29.309096 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.309072    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ace43d0-fe92-4e5e-a190-26408d4ab47b-workload-certs\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2\" (UID: \"8ace43d0-fe92-4e5e-a190-26408d4ab47b\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2"
Nov 03 22:29:29.309303 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.309100    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67fzs\" (UniqueName: \"kubernetes.io/projected/8ace43d0-fe92-4e5e-a190-26408d4ab47b-kube-api-access-67fzs\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2\" (UID: \"8ace43d0-fe92-4e5e-a190-26408d4ab47b\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2"
Nov 03 22:29:29.309303 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.309140    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ace43d0-fe92-4e5e-a190-26408d4ab47b-istio-token\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2\" (UID: \"8ace43d0-fe92-4e5e-a190-26408d4ab47b\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2"
Nov 03 22:29:29.309303 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.309237    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ace43d0-fe92-4e5e-a190-26408d4ab47b-credential-socket\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2\" (UID: \"8ace43d0-fe92-4e5e-a190-26408d4ab47b\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2"
Nov 03 22:29:29.309520 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.309424    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ace43d0-fe92-4e5e-a190-26408d4ab47b-istio-data\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2\" (UID: \"8ace43d0-fe92-4e5e-a190-26408d4ab47b\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2"
Nov 03 22:29:29.309625 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.309570    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ace43d0-fe92-4e5e-a190-26408d4ab47b-istiod-ca-cert\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2\" (UID: \"8ace43d0-fe92-4e5e-a190-26408d4ab47b\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2"
Nov 03 22:29:29.309625 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.309587    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ace43d0-fe92-4e5e-a190-26408d4ab47b-workload-certs\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2\" (UID: \"8ace43d0-fe92-4e5e-a190-26408d4ab47b\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2"
Nov 03 22:29:29.309776 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.309627    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ace43d0-fe92-4e5e-a190-26408d4ab47b-credential-socket\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2\" (UID: \"8ace43d0-fe92-4e5e-a190-26408d4ab47b\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2"
Nov 03 22:29:29.309852 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.309826    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ace43d0-fe92-4e5e-a190-26408d4ab47b-workload-socket\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2\" (UID: \"8ace43d0-fe92-4e5e-a190-26408d4ab47b\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2"
Nov 03 22:29:29.311945 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.311918    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ace43d0-fe92-4e5e-a190-26408d4ab47b-istio-envoy\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2\" (UID: \"8ace43d0-fe92-4e5e-a190-26408d4ab47b\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2"
Nov 03 22:29:29.312182 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.312163    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ace43d0-fe92-4e5e-a190-26408d4ab47b-istio-podinfo\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2\" (UID: \"8ace43d0-fe92-4e5e-a190-26408d4ab47b\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2"
Nov 03 22:29:29.315064 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.315043    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t"
Nov 03 22:29:29.340900 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.340691    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ace43d0-fe92-4e5e-a190-26408d4ab47b-istio-token\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2\" (UID: \"8ace43d0-fe92-4e5e-a190-26408d4ab47b\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2"
Nov 03 22:29:29.340900 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.340847    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67fzs\" (UniqueName: \"kubernetes.io/projected/8ace43d0-fe92-4e5e-a190-26408d4ab47b-kube-api-access-67fzs\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2\" (UID: \"8ace43d0-fe92-4e5e-a190-26408d4ab47b\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2"
Nov 03 22:29:29.384656 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.384567    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2"
Nov 03 22:29:29.411962 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.411929    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr"]
Nov 03 22:29:29.419325 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.419283    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:29:29.419434 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.419374    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:29:29.419434 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.419419    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:29:29.574579 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:29:29.574534    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36b9d353_1537_445b_bb03_d2d917a2a208.slice/crio-56f6e271ed6e1101bd73f614274476fb460b6ab669483ddefff55b9a623aaf30 WatchSource:0}: Error finding container 56f6e271ed6e1101bd73f614274476fb460b6ab669483ddefff55b9a623aaf30: Status 404 returned error can't find the container with id 56f6e271ed6e1101bd73f614274476fb460b6ab669483ddefff55b9a623aaf30
Nov 03 22:29:29.576077 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.574957    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc"]
Nov 03 22:29:29.580633 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.579747    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:29:29.580633 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.579834    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:29:29.580633 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.579879    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:29:29.656379 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.656350    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t"]
Nov 03 22:29:29.657921 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:29:29.657885    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3d85fcc_b84d_48a3_944a_810ce94062a2.slice/crio-e39377b09b5364991d9dafeb33fdcfb58084089c35b348170ef5a4602670275c WatchSource:0}: Error finding container e39377b09b5364991d9dafeb33fdcfb58084089c35b348170ef5a4602670275c: Status 404 returned error can't find the container with id e39377b09b5364991d9dafeb33fdcfb58084089c35b348170ef5a4602670275c
Nov 03 22:29:29.661486 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.661449    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:29:29.661563 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.661538    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:29:29.661609 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.661582    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:29:29.733368 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.731498    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2"]
Nov 03 22:29:29.737152 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:29:29.735264    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ace43d0_fe92_4e5e_a190_26408d4ab47b.slice/crio-0fb918dd53bfd27fa404b9570b969fa8212a467e7a59d47fb0eb33bff7b283c3 WatchSource:0}: Error finding container 0fb918dd53bfd27fa404b9570b969fa8212a467e7a59d47fb0eb33bff7b283c3: Status 404 returned error can't find the container with id 0fb918dd53bfd27fa404b9570b969fa8212a467e7a59d47fb0eb33bff7b283c3
Nov 03 22:29:29.739090 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.739051    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:29:29.739194 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.739143    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:29:29.739194 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:29.739185    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:29:30.401969 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:30.401929    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t" event={"ID":"e3d85fcc-b84d-48a3-944a-810ce94062a2","Type":"ContainerStarted","Data":"d10817a4ba8e04585139df549ff3a090c6dcbce256e774d93b060b76fb534f92"}
Nov 03 22:29:30.402493 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:30.401993    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t" event={"ID":"e3d85fcc-b84d-48a3-944a-810ce94062a2","Type":"ContainerStarted","Data":"e39377b09b5364991d9dafeb33fdcfb58084089c35b348170ef5a4602670275c"}
Nov 03 22:29:30.404480 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:30.404452    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr" event={"ID":"ffc24874-f9cc-4433-a4f4-2a72788b1067","Type":"ContainerStarted","Data":"655e474eb3e10ed5bd9595ca30aab9315af6f3fd42c43190a492d94f93d13df8"}
Nov 03 22:29:30.404606 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:30.404504    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr" event={"ID":"ffc24874-f9cc-4433-a4f4-2a72788b1067","Type":"ContainerStarted","Data":"1b62939e106025b1f179bc95ca68191991a0f31d69f592ee5c1be28e2be807fc"}
Nov 03 22:29:30.406617 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:30.406597    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2" event={"ID":"8ace43d0-fe92-4e5e-a190-26408d4ab47b","Type":"ContainerStarted","Data":"249324c1e5ce56378c577f6b78a20dc22cbe2aeffc0d5624b4e10a8e9b73399c"}
Nov 03 22:29:30.406680 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:30.406633    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2" event={"ID":"8ace43d0-fe92-4e5e-a190-26408d4ab47b","Type":"ContainerStarted","Data":"0fb918dd53bfd27fa404b9570b969fa8212a467e7a59d47fb0eb33bff7b283c3"}
Nov 03 22:29:30.409261 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:30.409243    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc" event={"ID":"36b9d353-1537-445b-bb03-d2d917a2a208","Type":"ContainerStarted","Data":"f36a69f53131d660f437b23e66abcb043c7a3e3f24581ccdf660bc3cb05c4789"}
Nov 03 22:29:30.409315 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:30.409276    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc" event={"ID":"36b9d353-1537-445b-bb03-d2d917a2a208","Type":"ContainerStarted","Data":"56f6e271ed6e1101bd73f614274476fb460b6ab669483ddefff55b9a623aaf30"}
Nov 03 22:29:30.472270 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:30.464426    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t" podStartSLOduration=2.464405825 podStartE2EDuration="2.464405825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:29:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:29:30.462607999 +0000 UTC m=+994.488463861" watchObservedRunningTime="2026-11-03 22:29:30.464405825 +0000 UTC m=+994.490261688"
Nov 03 22:29:30.514584 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:30.514512    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr" podStartSLOduration=2.514492466 podStartE2EDuration="2.514492466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:29:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:29:30.509953284 +0000 UTC m=+994.535809147" watchObservedRunningTime="2026-11-03 22:29:30.514492466 +0000 UTC m=+994.540348330"
Nov 03 22:29:30.560247 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:30.559205    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2" podStartSLOduration=2.559185292 podStartE2EDuration="2.559185292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:29:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:29:30.557801771 +0000 UTC m=+994.583657633" watchObservedRunningTime="2026-11-03 22:29:30.559185292 +0000 UTC m=+994.585041155"
Nov 03 22:29:30.629934 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:30.625117    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc" podStartSLOduration=2.6250930869999998 podStartE2EDuration="2.625093087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:29:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:29:30.608934898 +0000 UTC m=+994.634790773" watchObservedRunningTime="2026-11-03 22:29:30.625093087 +0000 UTC m=+994.650948950"
Nov 03 22:29:30.692251 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:30.689633    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:29:31.079301 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:31.078195    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr"
Nov 03 22:29:31.083243 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:31.083196    4431 patch_prober.go:29] interesting pod/e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 03 22:29:31.083537 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:31.083508    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr" podUID="ffc24874-f9cc-4433-a4f4-2a72788b1067" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 03 22:29:31.298501 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:31.298424    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc"
Nov 03 22:29:31.300429 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:31.300398    4431 patch_prober.go:29] interesting pod/e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 03 22:29:31.300689 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:31.300659    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc" podUID="36b9d353-1537-445b-bb03-d2d917a2a208" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 03 22:29:31.319589 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:31.319503    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t"
Nov 03 22:29:31.327238 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:31.324356    4431 patch_prober.go:29] interesting pod/e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 03 22:29:31.327238 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:31.324416    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t" podUID="e3d85fcc-b84d-48a3-944a-810ce94062a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 03 22:29:31.472008 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:31.433571    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2"
Nov 03 22:29:31.493280 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:31.490384    4431 patch_prober.go:29] interesting pod/e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 03 22:29:31.493280 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:31.490453    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2" podUID="8ace43d0-fe92-4e5e-a190-26408d4ab47b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 03 22:29:31.514257 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:29:31.501385    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:29:31.628349 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:31.628302    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-50d738/distinct-w4-50d738-kserve-668fdf8b86-5pp9p" podUID="ce50af62-9cb6-4379-99c7-b40630538f4f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:31.692357 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:31.692305    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-553753/distinct-2-w2-553753-kserve-5bf8568844-7q4l2" podUID="a1894b80-164c-4ce1-94f3-1e5011d9759e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:31.893299 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:31.890152    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:29:32.072240 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:32.057962    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:29:32.072240 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:32.067650    4431 patch_prober.go:29] interesting pod/e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 03 22:29:32.072240 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:32.067723    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr" podUID="ffc24874-f9cc-4433-a4f4-2a72788b1067" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 03 22:29:32.106029 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:32.105982    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/premium-w0-88a4d9-kserve-5559d75b7b-vqllq" podUID="05206737-4ca3-41fc-b35a-37c625cc182c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:32.214380 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:32.214343    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:29:32.298249 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:32.297355    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-w6-2d619f-kserve-74dbc56695-tnvtb" podUID="90310d72-c818-4fdd-b4e7-b51f633caa05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:32.298249 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:32.297694    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-50d738/premium-w4-50d738-kserve-59ccd7f995-xbfq2" podUID="fa0a680b-8f0d-4b19-8217-82282764511a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:32.345931 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:32.345891    4431 patch_prober.go:29] interesting pod/e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 03 22:29:32.346208 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:32.346177    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t" podUID="e3d85fcc-b84d-48a3-944a-810ce94062a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 03 22:29:32.347344 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:32.347319    4431 patch_prober.go:29] interesting pod/e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 03 22:29:32.347501 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:32.347482    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc" podUID="36b9d353-1537-445b-bb03-d2d917a2a208" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 03 22:29:32.393380 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:32.393314    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5187e1/distinct-w3-5187e1-kserve-545b7fb6bc-mk49g" podUID="3ba8dc17-1b1b-4811-9dd6-595d75c6ca04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:32.424239 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:32.406344    4431 patch_prober.go:29] interesting pod/e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 03 22:29:32.424239 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:32.406419    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2" podUID="8ace43d0-fe92-4e5e-a190-26408d4ab47b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 03 22:29:32.446241 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:32.437410    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-553753/simulator-w2-553753-kserve-64d77d4dfc-4rlbn" podUID="a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:32.446241 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:32.438163    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5187e1/premium-w3-5187e1-kserve-56bf8788d4-pp8st" podUID="a70ecfe9-020a-4331-b67c-aaaa46ed1eb1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Nov 03 22:29:32.473394 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:32.473289    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/simulator-w0-88a4d9-kserve-785f84888f-sjbzp" podUID="18bd460b-a0f1-415c-bcb4-41c9f70d2926" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:32.516471 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:32.516431    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-50d738/distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r" podUID="e45b1572-6519-4c71-884a-115eeddd86cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:32.674519 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:32.674462    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-50d738/distinct-w4-50d738-kserve-668fdf8b86-5pp9p" podUID="ce50af62-9cb6-4379-99c7-b40630538f4f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:32.675428 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:32.675407    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-50d738/distinct-w4-50d738-kserve-668fdf8b86-5pp9p"
Nov 03 22:29:32.711980 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:32.711906    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-553753/distinct-2-w2-553753-kserve-5bf8568844-7q4l2" podUID="a1894b80-164c-4ce1-94f3-1e5011d9759e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Nov 03 22:29:32.712536 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:32.712500    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-553753/distinct-2-w2-553753-kserve-5bf8568844-7q4l2"
Nov 03 22:29:32.806257 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:32.802096    4431 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5187e1/distinct-w3-5187e1-kserve-545b7fb6bc-mk49g" podUID="3ba8dc17-1b1b-4811-9dd6-595d75c6ca04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:33.250616 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:33.232773    4431 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-553753/distinct-2-w2-553753-kserve-5bf8568844-7q4l2" podUID="a1894b80-164c-4ce1-94f3-1e5011d9759e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:33.297247 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:33.276835    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/premium-w0-88a4d9-kserve-5559d75b7b-vqllq" podUID="05206737-4ca3-41fc-b35a-37c625cc182c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:33.351799 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:33.329277    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-50d738/premium-w4-50d738-kserve-59ccd7f995-xbfq2" podUID="fa0a680b-8f0d-4b19-8217-82282764511a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:33.351799 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:33.350517    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-88a4d9/premium-w0-88a4d9-kserve-5559d75b7b-vqllq"
Nov 03 22:29:33.351799 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:33.350580    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-50d738/premium-w4-50d738-kserve-59ccd7f995-xbfq2"
Nov 03 22:29:33.351799 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:33.329316    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-w6-2d619f-kserve-74dbc56695-tnvtb" podUID="90310d72-c818-4fdd-b4e7-b51f633caa05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:33.352204 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:33.330479    4431 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-50d738/premium-w4-50d738-kserve-59ccd7f995-xbfq2" podUID="fa0a680b-8f0d-4b19-8217-82282764511a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": context deadline exceeded"
Nov 03 22:29:33.361295 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:33.352764    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2d619f/distinct-w6-2d619f-kserve-74dbc56695-tnvtb"
Nov 03 22:29:33.554909 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:33.550037    4431 patch_prober.go:29] interesting pod/e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 03 22:29:33.554909 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:33.550094    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t" podUID="e3d85fcc-b84d-48a3-944a-810ce94062a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 03 22:29:33.554909 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:33.550366    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5187e1/premium-w3-5187e1-kserve-56bf8788d4-pp8st" podUID="a70ecfe9-020a-4331-b67c-aaaa46ed1eb1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:33.554909 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:33.550540    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-553753/simulator-w2-553753-kserve-64d77d4dfc-4rlbn" podUID="a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:33.554909 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:33.550722    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5187e1/premium-w3-5187e1-kserve-56bf8788d4-pp8st"
Nov 03 22:29:33.554909 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:33.550779    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-553753/simulator-w2-553753-kserve-64d77d4dfc-4rlbn"
Nov 03 22:29:33.602255 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:33.558272    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5187e1/distinct-w3-5187e1-kserve-545b7fb6bc-mk49g" podUID="3ba8dc17-1b1b-4811-9dd6-595d75c6ca04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:33.602255 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:33.569763    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-50d738/distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r" podUID="e45b1572-6519-4c71-884a-115eeddd86cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:33.602255 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:33.570567    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5187e1/distinct-w3-5187e1-kserve-545b7fb6bc-mk49g"
Nov 03 22:29:33.602255 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:33.570624    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-50d738/distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r"
Nov 03 22:29:33.680798 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:33.680744    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-50d738/distinct-w4-50d738-kserve-668fdf8b86-5pp9p" podUID="ce50af62-9cb6-4379-99c7-b40630538f4f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:33.735285 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:33.724369    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-553753/distinct-2-w2-553753-kserve-5bf8568844-7q4l2" podUID="a1894b80-164c-4ce1-94f3-1e5011d9759e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:33.768587 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:33.768555    4431 patch_prober.go:29] interesting pod/e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 03 22:29:33.768893 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:33.768851    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc" podUID="36b9d353-1537-445b-bb03-d2d917a2a208" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 03 22:29:33.795827 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:33.795778    4431 patch_prober.go:29] interesting pod/e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 03 22:29:33.796117 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:33.796088    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2" podUID="8ace43d0-fe92-4e5e-a190-26408d4ab47b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 03 22:29:33.908293 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:33.863588    4431 patch_prober.go:29] interesting pod/e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 03 22:29:34.174839 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:34.154013    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr" podUID="ffc24874-f9cc-4433-a4f4-2a72788b1067" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 03 22:29:34.178016 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:33.981376    4431 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-mrr4d" podUID="85a4d227-3a7e-4270-aabe-3a79d7f51c1d" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.21:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:34.230241 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:34.185141    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:29:34.289661 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:34.205601    4431 patch_prober.go:29] interesting pod/e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 03 22:29:34.289976 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:34.289942    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr" podUID="ffc24874-f9cc-4433-a4f4-2a72788b1067" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 03 22:29:34.357245 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:34.350334    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-50d738/premium-w4-50d738-kserve-59ccd7f995-xbfq2" podUID="fa0a680b-8f0d-4b19-8217-82282764511a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:34.373117 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:34.370912    4431 patch_prober.go:29] interesting pod/e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 03 22:29:34.373117 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:34.370966    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t" podUID="e3d85fcc-b84d-48a3-944a-810ce94062a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 03 22:29:34.373117 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:34.353680    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-w6-2d619f-kserve-74dbc56695-tnvtb" podUID="90310d72-c818-4fdd-b4e7-b51f633caa05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:34.373117 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:34.353638    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/premium-w0-88a4d9-kserve-5559d75b7b-vqllq" podUID="05206737-4ca3-41fc-b35a-37c625cc182c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 03 22:29:34.373117 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:34.353334    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-mrr4d" podUID="85a4d227-3a7e-4270-aabe-3a79d7f51c1d" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.21:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:34.373117 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:34.353712    4431 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-88a4d9/premium-w0-88a4d9-kserve-5559d75b7b-vqllq" podUID="05206737-4ca3-41fc-b35a-37c625cc182c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:34.653129 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:34.511764    4431 patch_prober.go:29] interesting pod/e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 03 22:29:34.653129 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:34.511826    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2" podUID="8ace43d0-fe92-4e5e-a190-26408d4ab47b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 03 22:29:34.653129 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:34.511909    4431 patch_prober.go:29] interesting pod/e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 03 22:29:34.653129 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:34.511932    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc" podUID="36b9d353-1537-445b-bb03-d2d917a2a208" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 03 22:29:34.653129 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:34.513351    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/simulator-w0-88a4d9-kserve-785f84888f-sjbzp" podUID="18bd460b-a0f1-415c-bcb4-41c9f70d2926" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:34.683736 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:34.673851    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-50d738/distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r" podUID="e45b1572-6519-4c71-884a-115eeddd86cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:34.684637 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:34.684595    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5187e1/premium-w3-5187e1-kserve-56bf8788d4-pp8st" podUID="a70ecfe9-020a-4331-b67c-aaaa46ed1eb1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:34.710640 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:34.685124    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5187e1/distinct-w3-5187e1-kserve-545b7fb6bc-mk49g" podUID="3ba8dc17-1b1b-4811-9dd6-595d75c6ca04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:34.712044 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:34.685152    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-553753/simulator-w2-553753-kserve-64d77d4dfc-4rlbn" podUID="a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:34.754253 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:34.724974    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-50d738/distinct-w4-50d738-kserve-668fdf8b86-5pp9p" podUID="ce50af62-9cb6-4379-99c7-b40630538f4f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:35.019241 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:34.989968    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-553753/distinct-2-w2-553753-kserve-5bf8568844-7q4l2"
Nov 03 22:29:35.073254 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:35.069978    4431 patch_prober.go:29] interesting pod/e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 03 22:29:35.073254 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:35.070032    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr" podUID="ffc24874-f9cc-4433-a4f4-2a72788b1067" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 03 22:29:35.296135 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:35.296030    4431 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-w6-2d619f-kserve-74dbc56695-tnvtb" podUID="90310d72-c818-4fdd-b4e7-b51f633caa05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:35.451337 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:35.307087    4431 patch_prober.go:29] interesting pod/e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 03 22:29:35.475365 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:35.475322    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc" podUID="36b9d353-1537-445b-bb03-d2d917a2a208" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 03 22:29:35.579418 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:35.400542    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-w6-2d619f-kserve-74dbc56695-tnvtb" podUID="90310d72-c818-4fdd-b4e7-b51f633caa05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:35.665251 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:35.400417    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/premium-w0-88a4d9-kserve-5559d75b7b-vqllq" podUID="05206737-4ca3-41fc-b35a-37c625cc182c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:35.665251 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:35.400495    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-50d738/premium-w4-50d738-kserve-59ccd7f995-xbfq2" podUID="fa0a680b-8f0d-4b19-8217-82282764511a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:35.665251 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:35.452450    4431 patch_prober.go:29] interesting pod/e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 03 22:29:35.665251 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:35.628567    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2" podUID="8ace43d0-fe92-4e5e-a190-26408d4ab47b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 03 22:29:35.665251 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:35.544567    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:29:35.665251 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:35.544350    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/simulator-w0-88a4d9-kserve-785f84888f-sjbzp" podUID="18bd460b-a0f1-415c-bcb4-41c9f70d2926" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded"
Nov 03 22:29:35.665251 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:35.467989    4431 patch_prober.go:29] interesting pod/e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 03 22:29:35.665251 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:35.628914    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t" podUID="e3d85fcc-b84d-48a3-944a-810ce94062a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 03 22:29:35.665251 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:35.597148    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-553753/distinct-2-w2-553753-kserve-5bf8568844-7q4l2" podUID="a1894b80-164c-4ce1-94f3-1e5011d9759e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:35.665251 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:35.629231    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-88a4d9/simulator-w0-88a4d9-kserve-785f84888f-sjbzp"
Nov 03 22:29:35.735237 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:35.733551    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-50d738/distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r" podUID="e45b1572-6519-4c71-884a-115eeddd86cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:35.735237 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:35.734285    4431 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-50d738/distinct-w4-50d738-kserve-668fdf8b86-5pp9p" podUID="ce50af62-9cb6-4379-99c7-b40630538f4f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:35.757268 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:35.752944    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5187e1/distinct-w3-5187e1-kserve-545b7fb6bc-mk49g" podUID="3ba8dc17-1b1b-4811-9dd6-595d75c6ca04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:35.765267 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:35.762852    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-50d738/distinct-w4-50d738-kserve-668fdf8b86-5pp9p" podUID="ce50af62-9cb6-4379-99c7-b40630538f4f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:35.765267 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:35.763873    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-553753/simulator-w2-553753-kserve-64d77d4dfc-4rlbn" podUID="a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Nov 03 22:29:35.806240 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:35.805287    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5187e1/premium-w3-5187e1-kserve-56bf8788d4-pp8st" podUID="a70ecfe9-020a-4331-b67c-aaaa46ed1eb1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Nov 03 22:29:36.143237 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:36.140350    4431 patch_prober.go:29] interesting pod/e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 03 22:29:36.143237 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:36.140418    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr" podUID="ffc24874-f9cc-4433-a4f4-2a72788b1067" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 03 22:29:36.298242 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:36.298157    4431 patch_prober.go:29] interesting pod/e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 03 22:29:36.325599 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:36.325535    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc" podUID="36b9d353-1537-445b-bb03-d2d917a2a208" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 03 22:29:36.370501 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:36.370463    4431 patch_prober.go:29] interesting pod/e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 03 22:29:36.370751 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:36.370722    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t" podUID="e3d85fcc-b84d-48a3-944a-810ce94062a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 03 22:29:36.446481 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:36.446439    4431 patch_prober.go:29] interesting pod/e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 03 22:29:36.469374 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:36.469323    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2" podUID="8ace43d0-fe92-4e5e-a190-26408d4ab47b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 03 22:29:36.655632 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:36.655558    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-w6-2d619f-kserve-74dbc56695-tnvtb" podUID="90310d72-c818-4fdd-b4e7-b51f633caa05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:36.726734 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:36.726638    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-50d738/premium-w4-50d738-kserve-59ccd7f995-xbfq2" podUID="fa0a680b-8f0d-4b19-8217-82282764511a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:36.727557 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:36.727528    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/premium-w0-88a4d9-kserve-5559d75b7b-vqllq" podUID="05206737-4ca3-41fc-b35a-37c625cc182c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 03 22:29:36.727979 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:36.727955    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-553753/distinct-2-w2-553753-kserve-5bf8568844-7q4l2" podUID="a1894b80-164c-4ce1-94f3-1e5011d9759e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:36.728357 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:36.728334    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/simulator-w0-88a4d9-kserve-785f84888f-sjbzp" podUID="18bd460b-a0f1-415c-bcb4-41c9f70d2926" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded"
Nov 03 22:29:36.738605 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:36.738578    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-553753/distinct-2-w2-553753-kserve-5bf8568844-7q4l2"
Nov 03 22:29:36.743535 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:36.743500    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-50d738/distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r" podUID="e45b1572-6519-4c71-884a-115eeddd86cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:36.753542 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:36.753505    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5187e1/distinct-w3-5187e1-kserve-545b7fb6bc-mk49g" podUID="3ba8dc17-1b1b-4811-9dd6-595d75c6ca04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:36.775064 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:36.774305    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-553753/simulator-w2-553753-kserve-64d77d4dfc-4rlbn" podUID="a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:36.775064 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:36.774707    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-50d738/distinct-w4-50d738-kserve-668fdf8b86-5pp9p" podUID="ce50af62-9cb6-4379-99c7-b40630538f4f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 03 22:29:36.819849 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:36.819807    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5187e1/premium-w3-5187e1-kserve-56bf8788d4-pp8st" podUID="a70ecfe9-020a-4331-b67c-aaaa46ed1eb1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:37.066612 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:37.066522    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-88a4d9/premium-w0-88a4d9-kserve-5559d75b7b-vqllq"
Nov 03 22:29:37.071242 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:37.067647    4431 patch_prober.go:29] interesting pod/e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 03 22:29:37.073867 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:37.073834    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr" podUID="ffc24874-f9cc-4433-a4f4-2a72788b1067" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 03 22:29:37.124510 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:37.124469    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-50d738/distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r"
Nov 03 22:29:37.236237 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:37.231458    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5187e1/distinct-w3-5187e1-kserve-545b7fb6bc-mk49g"
Nov 03 22:29:37.309259 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:37.308761    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-553753/simulator-w2-553753-kserve-64d77d4dfc-4rlbn"
Nov 03 22:29:37.311446 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:37.309699    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-88a4d9/simulator-w0-88a4d9-kserve-785f84888f-sjbzp"
Nov 03 22:29:37.329333 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:37.329206    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5187e1/premium-w3-5187e1-kserve-56bf8788d4-pp8st"
Nov 03 22:29:37.390267 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:37.335860    4431 patch_prober.go:29] interesting pod/e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 03 22:29:37.409596 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:37.409548    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t" podUID="e3d85fcc-b84d-48a3-944a-810ce94062a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 03 22:29:37.409853 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:37.387845    4431 patch_prober.go:29] interesting pod/e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 03 22:29:37.409970 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:37.409951    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2" podUID="8ace43d0-fe92-4e5e-a190-26408d4ab47b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 03 22:29:37.410070 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:37.388271    4431 patch_prober.go:29] interesting pod/e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 03 22:29:37.419483 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:37.410168    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc" podUID="36b9d353-1537-445b-bb03-d2d917a2a208" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 03 22:29:37.419703 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:37.414785    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2d619f/distinct-w6-2d619f-kserve-74dbc56695-tnvtb"
Nov 03 22:29:37.419889 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:37.419869    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-50d738/premium-w4-50d738-kserve-59ccd7f995-xbfq2"
Nov 03 22:29:37.461646 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:37.461613    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-50d738/distinct-w4-50d738-kserve-668fdf8b86-5pp9p"
Nov 03 22:29:37.468097 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:37.468064    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-553753/distinct-2-w2-553753-kserve-5bf8568844-7q4l2"
Nov 03 22:29:37.686543 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:37.685753    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:29:38.069689 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:38.069590    4431 patch_prober.go:29] interesting pod/e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 03 22:29:38.070297 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:38.070208    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr" podUID="ffc24874-f9cc-4433-a4f4-2a72788b1067" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 03 22:29:38.317596 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:38.317553    4431 patch_prober.go:29] interesting pod/e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 03 22:29:38.317888 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:38.317843    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t" podUID="e3d85fcc-b84d-48a3-944a-810ce94062a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 03 22:29:38.390779 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:38.387396    4431 patch_prober.go:29] interesting pod/e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 03 22:29:38.390779 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:38.387463    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2" podUID="8ace43d0-fe92-4e5e-a190-26408d4ab47b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 03 22:29:38.997371 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:38.997343    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc"
Nov 03 22:29:38.998536 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:38.998515    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc"
Nov 03 22:29:39.021469 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:39.021410    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc"
Nov 03 22:29:39.064922 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:39.064882    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:29:39.081118 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:39.081086    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr"
Nov 03 22:29:39.085285 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:39.085258    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr"
Nov 03 22:29:39.086271 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:39.086250    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr"
Nov 03 22:29:39.321191 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:39.321097    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t"
Nov 03 22:29:39.321516 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:39.321494    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t"
Nov 03 22:29:39.322454 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:39.322424    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t"
Nov 03 22:29:39.375056 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:39.375029    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-51bfa2e1-6954676576-kr8sq"
Nov 03 22:29:39.376889 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:39.376835    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:29:39.433258 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:39.433192    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2"
Nov 03 22:29:39.433741 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:39.433654    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2"
Nov 03 22:29:39.434593 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:39.434572    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2"
Nov 03 22:29:39.579124 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:39.579042    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:29:39.763084 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:39.763055    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:29:40.333961 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:40.333925    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:29:40.480013 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:40.479975    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:29:41.442579 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:41.442539    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:29:42.188589 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:42.188551    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:29:42.899241 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:42.898498    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg"]
Nov 03 22:29:42.905905 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:42.905874    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg"
Nov 03 22:29:42.933954 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:42.933925    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-53879c92-openshift-default-dockercfg-kcwdw\""
Nov 03 22:29:42.950994 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:42.950932    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg"]
Nov 03 22:29:42.953549 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:42.953209    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3221d3a-b814-43ec-a87a-831dd57c674d-istiod-ca-cert\") pod \"e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg\" (UID: \"e3221d3a-b814-43ec-a87a-831dd57c674d\") " pod="openshift-ingress/e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg"
Nov 03 22:29:42.953861 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:42.953826    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3221d3a-b814-43ec-a87a-831dd57c674d-istio-token\") pod \"e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg\" (UID: \"e3221d3a-b814-43ec-a87a-831dd57c674d\") " pod="openshift-ingress/e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg"
Nov 03 22:29:42.954162 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:42.954130    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3221d3a-b814-43ec-a87a-831dd57c674d-istio-podinfo\") pod \"e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg\" (UID: \"e3221d3a-b814-43ec-a87a-831dd57c674d\") " pod="openshift-ingress/e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg"
Nov 03 22:29:42.954361 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:42.954333    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3221d3a-b814-43ec-a87a-831dd57c674d-istio-data\") pod \"e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg\" (UID: \"e3221d3a-b814-43ec-a87a-831dd57c674d\") " pod="openshift-ingress/e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg"
Nov 03 22:29:42.954636 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:42.954602    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3221d3a-b814-43ec-a87a-831dd57c674d-istio-envoy\") pod \"e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg\" (UID: \"e3221d3a-b814-43ec-a87a-831dd57c674d\") " pod="openshift-ingress/e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg"
Nov 03 22:29:42.954802 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:42.954781    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3221d3a-b814-43ec-a87a-831dd57c674d-workload-certs\") pod \"e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg\" (UID: \"e3221d3a-b814-43ec-a87a-831dd57c674d\") " pod="openshift-ingress/e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg"
Nov 03 22:29:42.954867 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:42.954846    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3221d3a-b814-43ec-a87a-831dd57c674d-credential-socket\") pod \"e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg\" (UID: \"e3221d3a-b814-43ec-a87a-831dd57c674d\") " pod="openshift-ingress/e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg"
Nov 03 22:29:42.954924 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:42.954885    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2kp8\" (UniqueName: \"kubernetes.io/projected/e3221d3a-b814-43ec-a87a-831dd57c674d-kube-api-access-k2kp8\") pod \"e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg\" (UID: \"e3221d3a-b814-43ec-a87a-831dd57c674d\") " pod="openshift-ingress/e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg"
Nov 03 22:29:42.954985 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:42.954941    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3221d3a-b814-43ec-a87a-831dd57c674d-workload-socket\") pod \"e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg\" (UID: \"e3221d3a-b814-43ec-a87a-831dd57c674d\") " pod="openshift-ingress/e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg"
Nov 03 22:29:42.992429 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:42.992381    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:29:43.056499 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:43.056466    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3221d3a-b814-43ec-a87a-831dd57c674d-istiod-ca-cert\") pod \"e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg\" (UID: \"e3221d3a-b814-43ec-a87a-831dd57c674d\") " pod="openshift-ingress/e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg"
Nov 03 22:29:43.056677 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:43.056561    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3221d3a-b814-43ec-a87a-831dd57c674d-istio-token\") pod \"e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg\" (UID: \"e3221d3a-b814-43ec-a87a-831dd57c674d\") " pod="openshift-ingress/e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg"
Nov 03 22:29:43.056759 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:43.056714    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3221d3a-b814-43ec-a87a-831dd57c674d-istio-podinfo\") pod \"e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg\" (UID: \"e3221d3a-b814-43ec-a87a-831dd57c674d\") " pod="openshift-ingress/e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg"
Nov 03 22:29:43.056821 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:43.056787    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3221d3a-b814-43ec-a87a-831dd57c674d-istio-data\") pod \"e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg\" (UID: \"e3221d3a-b814-43ec-a87a-831dd57c674d\") " pod="openshift-ingress/e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg"
Nov 03 22:29:43.056873 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:43.056830    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3221d3a-b814-43ec-a87a-831dd57c674d-istio-envoy\") pod \"e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg\" (UID: \"e3221d3a-b814-43ec-a87a-831dd57c674d\") " pod="openshift-ingress/e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg"
Nov 03 22:29:43.056922 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:43.056899    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3221d3a-b814-43ec-a87a-831dd57c674d-workload-certs\") pod \"e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg\" (UID: \"e3221d3a-b814-43ec-a87a-831dd57c674d\") " pod="openshift-ingress/e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg"
Nov 03 22:29:43.056969 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:43.056945    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3221d3a-b814-43ec-a87a-831dd57c674d-credential-socket\") pod \"e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg\" (UID: \"e3221d3a-b814-43ec-a87a-831dd57c674d\") " pod="openshift-ingress/e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg"
Nov 03 22:29:43.057025 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:43.056986    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2kp8\" (UniqueName: \"kubernetes.io/projected/e3221d3a-b814-43ec-a87a-831dd57c674d-kube-api-access-k2kp8\") pod \"e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg\" (UID: \"e3221d3a-b814-43ec-a87a-831dd57c674d\") " pod="openshift-ingress/e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg"
Nov 03 22:29:43.057083 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:43.057044    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3221d3a-b814-43ec-a87a-831dd57c674d-workload-socket\") pod \"e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg\" (UID: \"e3221d3a-b814-43ec-a87a-831dd57c674d\") " pod="openshift-ingress/e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg"
Nov 03 22:29:43.057413 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:43.057387    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3221d3a-b814-43ec-a87a-831dd57c674d-istiod-ca-cert\") pod \"e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg\" (UID: \"e3221d3a-b814-43ec-a87a-831dd57c674d\") " pod="openshift-ingress/e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg"
Nov 03 22:29:43.058193 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:43.058161    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3221d3a-b814-43ec-a87a-831dd57c674d-istio-data\") pod \"e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg\" (UID: \"e3221d3a-b814-43ec-a87a-831dd57c674d\") " pod="openshift-ingress/e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg"
Nov 03 22:29:43.058310 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:43.058170    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3221d3a-b814-43ec-a87a-831dd57c674d-workload-certs\") pod \"e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg\" (UID: \"e3221d3a-b814-43ec-a87a-831dd57c674d\") " pod="openshift-ingress/e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg"
Nov 03 22:29:43.058610 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:43.058587    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3221d3a-b814-43ec-a87a-831dd57c674d-workload-socket\") pod \"e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg\" (UID: \"e3221d3a-b814-43ec-a87a-831dd57c674d\") " pod="openshift-ingress/e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg"
Nov 03 22:29:43.058886 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:43.058771    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3221d3a-b814-43ec-a87a-831dd57c674d-credential-socket\") pod \"e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg\" (UID: \"e3221d3a-b814-43ec-a87a-831dd57c674d\") " pod="openshift-ingress/e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg"
Nov 03 22:29:43.062903 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:43.062881    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3221d3a-b814-43ec-a87a-831dd57c674d-istio-envoy\") pod \"e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg\" (UID: \"e3221d3a-b814-43ec-a87a-831dd57c674d\") " pod="openshift-ingress/e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg"
Nov 03 22:29:43.086841 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:43.080072    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3221d3a-b814-43ec-a87a-831dd57c674d-istio-podinfo\") pod \"e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg\" (UID: \"e3221d3a-b814-43ec-a87a-831dd57c674d\") " pod="openshift-ingress/e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg"
Nov 03 22:29:43.088771 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:43.087407    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3221d3a-b814-43ec-a87a-831dd57c674d-istio-token\") pod \"e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg\" (UID: \"e3221d3a-b814-43ec-a87a-831dd57c674d\") " pod="openshift-ingress/e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg"
Nov 03 22:29:43.088771 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:43.087602    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2kp8\" (UniqueName: \"kubernetes.io/projected/e3221d3a-b814-43ec-a87a-831dd57c674d-kube-api-access-k2kp8\") pod \"e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg\" (UID: \"e3221d3a-b814-43ec-a87a-831dd57c674d\") " pod="openshift-ingress/e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg"
Nov 03 22:29:43.136914 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:43.136873    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:29:43.261313 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:43.260978    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg"
Nov 03 22:29:43.589480 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:43.589120    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg"]
Nov 03 22:29:43.594571 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:29:43.594509    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3221d3a_b814_43ec_a87a_831dd57c674d.slice/crio-dd48b41b565cf08be92bbdb5bee91937a77e5304741700de9ac2785e85114c4f WatchSource:0}: Error finding container dd48b41b565cf08be92bbdb5bee91937a77e5304741700de9ac2785e85114c4f: Status 404 returned error can't find the container with id dd48b41b565cf08be92bbdb5bee91937a77e5304741700de9ac2785e85114c4f
Nov 03 22:29:43.595494 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:43.595470    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:29:43.597065 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:43.597010    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:29:43.597286 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:43.597262    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:29:43.597393 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:43.597324    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:29:43.800647 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:43.800604    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg" event={"ID":"e3221d3a-b814-43ec-a87a-831dd57c674d","Type":"ContainerStarted","Data":"185a55c0e354b77ac26fb94856f91ac51983c791c8230390f953571f7de2b8df"}
Nov 03 22:29:43.800835 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:43.800660    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg" event={"ID":"e3221d3a-b814-43ec-a87a-831dd57c674d","Type":"ContainerStarted","Data":"dd48b41b565cf08be92bbdb5bee91937a77e5304741700de9ac2785e85114c4f"}
Nov 03 22:29:43.871474 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:43.871368    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg" podStartSLOduration=1.8713535700000001 podStartE2EDuration="1.87135357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:29:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:29:43.867521135 +0000 UTC m=+1007.893376999" watchObservedRunningTime="2026-11-03 22:29:43.87135357 +0000 UTC m=+1007.897209431"
Nov 03 22:29:44.056243 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:44.039861    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk"]
Nov 03 22:29:44.086665 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:44.082777    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk"
Nov 03 22:29:44.106271 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:44.104916    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk"]
Nov 03 22:29:44.188117 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:44.188082    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b92034e4-f537-425b-b28b-1cd528b700af-istio-token\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk\" (UID: \"b92034e4-f537-425b-b28b-1cd528b700af\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk"
Nov 03 22:29:44.188314 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:44.188143    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b92034e4-f537-425b-b28b-1cd528b700af-istio-envoy\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk\" (UID: \"b92034e4-f537-425b-b28b-1cd528b700af\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk"
Nov 03 22:29:44.188314 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:44.188171    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b92034e4-f537-425b-b28b-1cd528b700af-istio-data\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk\" (UID: \"b92034e4-f537-425b-b28b-1cd528b700af\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk"
Nov 03 22:29:44.188471 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:44.188394    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b92034e4-f537-425b-b28b-1cd528b700af-workload-certs\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk\" (UID: \"b92034e4-f537-425b-b28b-1cd528b700af\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk"
Nov 03 22:29:44.188471 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:44.188450    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4gcw\" (UniqueName: \"kubernetes.io/projected/b92034e4-f537-425b-b28b-1cd528b700af-kube-api-access-t4gcw\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk\" (UID: \"b92034e4-f537-425b-b28b-1cd528b700af\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk"
Nov 03 22:29:44.188576 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:44.188507    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b92034e4-f537-425b-b28b-1cd528b700af-credential-socket\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk\" (UID: \"b92034e4-f537-425b-b28b-1cd528b700af\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk"
Nov 03 22:29:44.188639 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:44.188620    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b92034e4-f537-425b-b28b-1cd528b700af-workload-socket\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk\" (UID: \"b92034e4-f537-425b-b28b-1cd528b700af\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk"
Nov 03 22:29:44.188705 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:44.188689    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b92034e4-f537-425b-b28b-1cd528b700af-istiod-ca-cert\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk\" (UID: \"b92034e4-f537-425b-b28b-1cd528b700af\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk"
Nov 03 22:29:44.189063 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:44.189038    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b92034e4-f537-425b-b28b-1cd528b700af-istio-podinfo\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk\" (UID: \"b92034e4-f537-425b-b28b-1cd528b700af\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk"
Nov 03 22:29:44.262150 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:44.262115    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg"
Nov 03 22:29:44.268702 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:44.268668    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg"
Nov 03 22:29:44.290004 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:44.289975    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b92034e4-f537-425b-b28b-1cd528b700af-istio-envoy\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk\" (UID: \"b92034e4-f537-425b-b28b-1cd528b700af\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk"
Nov 03 22:29:44.290004 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:44.290005    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b92034e4-f537-425b-b28b-1cd528b700af-istio-data\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk\" (UID: \"b92034e4-f537-425b-b28b-1cd528b700af\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk"
Nov 03 22:29:44.290258 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:44.290060    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b92034e4-f537-425b-b28b-1cd528b700af-workload-certs\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk\" (UID: \"b92034e4-f537-425b-b28b-1cd528b700af\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk"
Nov 03 22:29:44.290258 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:44.290098    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4gcw\" (UniqueName: \"kubernetes.io/projected/b92034e4-f537-425b-b28b-1cd528b700af-kube-api-access-t4gcw\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk\" (UID: \"b92034e4-f537-425b-b28b-1cd528b700af\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk"
Nov 03 22:29:44.290258 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:44.290128    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b92034e4-f537-425b-b28b-1cd528b700af-credential-socket\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk\" (UID: \"b92034e4-f537-425b-b28b-1cd528b700af\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk"
Nov 03 22:29:44.290258 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:44.290193    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b92034e4-f537-425b-b28b-1cd528b700af-workload-socket\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk\" (UID: \"b92034e4-f537-425b-b28b-1cd528b700af\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk"
Nov 03 22:29:44.290258 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:44.290235    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b92034e4-f537-425b-b28b-1cd528b700af-istiod-ca-cert\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk\" (UID: \"b92034e4-f537-425b-b28b-1cd528b700af\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk"
Nov 03 22:29:44.290562 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:44.290391    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b92034e4-f537-425b-b28b-1cd528b700af-istio-podinfo\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk\" (UID: \"b92034e4-f537-425b-b28b-1cd528b700af\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk"
Nov 03 22:29:44.290562 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:44.290509    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b92034e4-f537-425b-b28b-1cd528b700af-istio-token\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk\" (UID: \"b92034e4-f537-425b-b28b-1cd528b700af\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk"
Nov 03 22:29:44.290965 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:44.290937    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b92034e4-f537-425b-b28b-1cd528b700af-istiod-ca-cert\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk\" (UID: \"b92034e4-f537-425b-b28b-1cd528b700af\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk"
Nov 03 22:29:44.291251 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:44.291154    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b92034e4-f537-425b-b28b-1cd528b700af-workload-certs\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk\" (UID: \"b92034e4-f537-425b-b28b-1cd528b700af\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk"
Nov 03 22:29:44.291327 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:44.291270    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b92034e4-f537-425b-b28b-1cd528b700af-istio-data\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk\" (UID: \"b92034e4-f537-425b-b28b-1cd528b700af\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk"
Nov 03 22:29:44.291327 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:44.291309    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b92034e4-f537-425b-b28b-1cd528b700af-credential-socket\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk\" (UID: \"b92034e4-f537-425b-b28b-1cd528b700af\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk"
Nov 03 22:29:44.291470 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:44.291452    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b92034e4-f537-425b-b28b-1cd528b700af-workload-socket\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk\" (UID: \"b92034e4-f537-425b-b28b-1cd528b700af\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk"
Nov 03 22:29:44.292874 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:44.292853    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b92034e4-f537-425b-b28b-1cd528b700af-istio-podinfo\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk\" (UID: \"b92034e4-f537-425b-b28b-1cd528b700af\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk"
Nov 03 22:29:44.292949 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:44.292858    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b92034e4-f537-425b-b28b-1cd528b700af-istio-envoy\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk\" (UID: \"b92034e4-f537-425b-b28b-1cd528b700af\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk"
Nov 03 22:29:44.313305 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:44.313279    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b92034e4-f537-425b-b28b-1cd528b700af-istio-token\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk\" (UID: \"b92034e4-f537-425b-b28b-1cd528b700af\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk"
Nov 03 22:29:44.314094 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:44.313799    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4gcw\" (UniqueName: \"kubernetes.io/projected/b92034e4-f537-425b-b28b-1cd528b700af-kube-api-access-t4gcw\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk\" (UID: \"b92034e4-f537-425b-b28b-1cd528b700af\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk"
Nov 03 22:29:44.416245 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:44.416177    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk"
Nov 03 22:29:44.686684 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:44.686649    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:29:44.752502 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:44.752471    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk"]
Nov 03 22:29:44.754009 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:29:44.753973    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb92034e4_f537_425b_b28b_1cd528b700af.slice/crio-42cbe28309d690fa5bec84534d4b3aa5b282ea11ec4d4cdd462b01988af14aab WatchSource:0}: Error finding container 42cbe28309d690fa5bec84534d4b3aa5b282ea11ec4d4cdd462b01988af14aab: Status 404 returned error can't find the container with id 42cbe28309d690fa5bec84534d4b3aa5b282ea11ec4d4cdd462b01988af14aab
Nov 03 22:29:44.756993 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:44.756960    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:29:44.757079 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:44.757024    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:29:44.757079 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:44.757052    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:29:44.810614 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:44.810581    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk" event={"ID":"b92034e4-f537-425b-b28b-1cd528b700af","Type":"ContainerStarted","Data":"42cbe28309d690fa5bec84534d4b3aa5b282ea11ec4d4cdd462b01988af14aab"}
Nov 03 22:29:44.810886 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:44.810867    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg"
Nov 03 22:29:44.811764 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:44.811746    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg"
Nov 03 22:29:44.826490 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:44.826460    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:29:45.819081 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:45.819044    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk" event={"ID":"b92034e4-f537-425b-b28b-1cd528b700af","Type":"ContainerStarted","Data":"c67ac168dfce0b0498f3b2d6508b0683ebf6181f618986efd1eebcdf46ddb89c"}
Nov 03 22:29:45.892098 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:45.892040    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk" podStartSLOduration=2.892017757 podStartE2EDuration="2.892017757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:29:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:29:45.875472324 +0000 UTC m=+1009.901328185" watchObservedRunningTime="2026-11-03 22:29:45.892017757 +0000 UTC m=+1009.917873618"
Nov 03 22:29:46.021573 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:46.021531    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:29:46.333894 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:46.333856    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:29:46.417131 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:46.417102    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk"
Nov 03 22:29:46.418299 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:46.418274    4431 patch_prober.go:29] interesting pod/e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 03 22:29:46.418365 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:46.418327    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk" podUID="b92034e4-f537-425b-b28b-1cd528b700af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 03 22:29:46.600905 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:46.600813    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:29:46.915067 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:46.915035    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:29:47.079339 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:47.079295    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:29:47.416832 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:47.416793    4431 patch_prober.go:29] interesting pod/e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 03 22:29:47.417001 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:47.416848    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk" podUID="b92034e4-f537-425b-b28b-1cd528b700af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 03 22:29:47.971495 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:47.971454    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:29:48.128112 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:48.128075    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:29:48.417317 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:48.417285    4431 patch_prober.go:29] interesting pod/e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 03 22:29:48.417518 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:48.417346    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk" podUID="b92034e4-f537-425b-b28b-1cd528b700af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 03 22:29:48.859301 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:48.855879    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:29:49.417233 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:49.417183    4431 patch_prober.go:29] interesting pod/e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 03 22:29:49.417703 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:49.417278    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk" podUID="b92034e4-f537-425b-b28b-1cd528b700af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 03 22:29:49.776897 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:49.776780    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:29:49.942627 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:49.941302    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:29:50.323752 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:50.323713    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b7b10505-7645b7c844-knzm7"]
Nov 03 22:29:50.324409 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:50.324376    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b7b10505-7645b7c844-knzm7" podUID="6665b06d-4884-4946-947c-cbbd693d58c9" containerName="maas-api" containerID="cri-o://8e7e11d6419b51f321e3da9ef2ae85a75d5d7956f1e9af9dd20c07d228744a1c" gracePeriod=30
Nov 03 22:29:50.432301 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:50.432241    4431 patch_prober.go:29] interesting pod/e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 03 22:29:50.435102 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:50.435069    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk" podUID="b92034e4-f537-425b-b28b-1cd528b700af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 03 22:29:50.435609 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:50.433246    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:29:51.065792 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:51.065657    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:29:51.182643 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:51.182603    4431 generic.go:356] "Generic (PLEG): container finished" podID="6665b06d-4884-4946-947c-cbbd693d58c9" containerID="8e7e11d6419b51f321e3da9ef2ae85a75d5d7956f1e9af9dd20c07d228744a1c" exitCode=0
Nov 03 22:29:51.183802 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:51.183774    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b7b10505-7645b7c844-knzm7" event={"ID":"6665b06d-4884-4946-947c-cbbd693d58c9","Type":"ContainerDied","Data":"8e7e11d6419b51f321e3da9ef2ae85a75d5d7956f1e9af9dd20c07d228744a1c"}
Nov 03 22:29:51.227384 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:51.226118    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:29:51.374283 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:51.374102    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:29:51.420247 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:51.419792    4431 patch_prober.go:29] interesting pod/e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 03 22:29:51.420247 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:51.419850    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk" podUID="b92034e4-f537-425b-b28b-1cd528b700af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 03 22:29:51.537699 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:51.537594    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:29:52.180614 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:52.180566    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b7b10505-7645b7c844-knzm7"
Nov 03 22:29:52.223837 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:52.206874    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:29:52.282760 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:52.282708    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b7b10505-7645b7c844-knzm7" event={"ID":"6665b06d-4884-4946-947c-cbbd693d58c9","Type":"ContainerDied","Data":"03551a9c6f0d9c29a83d492fc229f34802fd57c1d05b0c5cd1d6271238ba66c6"}
Nov 03 22:29:52.283084 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:52.283047    4431 scope.go:122] "RemoveContainer" containerID="8e7e11d6419b51f321e3da9ef2ae85a75d5d7956f1e9af9dd20c07d228744a1c"
Nov 03 22:29:52.283499 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:52.283477    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b7b10505-7645b7c844-knzm7"
Nov 03 22:29:52.395241 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:52.394457    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6665b06d-4884-4946-947c-cbbd693d58c9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6665b06d-4884-4946-947c-cbbd693d58c9-maas-api-tls\") pod \"6665b06d-4884-4946-947c-cbbd693d58c9\" (UID: \"6665b06d-4884-4946-947c-cbbd693d58c9\") "
Nov 03 22:29:52.395241 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:52.394600    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6665b06d-4884-4946-947c-cbbd693d58c9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6665b06d-4884-4946-947c-cbbd693d58c9-metrics-certs\") pod \"6665b06d-4884-4946-947c-cbbd693d58c9\" (UID: \"6665b06d-4884-4946-947c-cbbd693d58c9\") "
Nov 03 22:29:52.395241 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:52.394746    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6665b06d-4884-4946-947c-cbbd693d58c9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6665b06d-4884-4946-947c-cbbd693d58c9-openshift-service-ca\") pod \"6665b06d-4884-4946-947c-cbbd693d58c9\" (UID: \"6665b06d-4884-4946-947c-cbbd693d58c9\") "
Nov 03 22:29:52.442091 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:52.396315    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-50d738/premium-w4-50d738-kserve-59ccd7f995-xbfq2" podUID="fa0a680b-8f0d-4b19-8217-82282764511a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:52.442091 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:52.416255    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6665b06d-4884-4946-947c-cbbd693d58c9-kube-api-access-djhdr\" (UniqueName: \"kubernetes.io/projected/6665b06d-4884-4946-947c-cbbd693d58c9-kube-api-access-djhdr\") pod \"6665b06d-4884-4946-947c-cbbd693d58c9\" (UID: \"6665b06d-4884-4946-947c-cbbd693d58c9\") "
Nov 03 22:29:52.442091 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:52.416939    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-w6-2d619f-kserve-74dbc56695-tnvtb" podUID="90310d72-c818-4fdd-b4e7-b51f633caa05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:52.442091 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:52.419827    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5187e1/premium-w3-5187e1-kserve-56bf8788d4-pp8st" podUID="a70ecfe9-020a-4331-b67c-aaaa46ed1eb1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:52.442091 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:52.420043    4431 patch_prober.go:29] interesting pod/e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 03 22:29:52.442091 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:52.420080    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk" podUID="b92034e4-f537-425b-b28b-1cd528b700af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 03 22:29:52.473318 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:52.472464    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6665b06d-4884-4946-947c-cbbd693d58c9-openshift-service-ca" pod "6665b06d-4884-4946-947c-cbbd693d58c9" (UID: "6665b06d-4884-4946-947c-cbbd693d58c9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:29:52.476276 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:52.476170    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-553753/distinct-2-w2-553753-kserve-5bf8568844-7q4l2" podUID="a1894b80-164c-4ce1-94f3-1e5011d9759e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:52.527238 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:52.525189    4431 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6665b06d-4884-4946-947c-cbbd693d58c9-openshift-service-ca\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:29:52.691245 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:52.690470    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6665b06d-4884-4946-947c-cbbd693d58c9-kube-api-access-djhdr" pod "6665b06d-4884-4946-947c-cbbd693d58c9" (UID: "6665b06d-4884-4946-947c-cbbd693d58c9"). InnerVolumeSpecName "kube-api-access-djhdr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:29:52.691711 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:52.691450    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/premium-w0-88a4d9-kserve-5559d75b7b-vqllq" podUID="05206737-4ca3-41fc-b35a-37c625cc182c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:52.718241 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:52.715625    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5187e1/distinct-w3-5187e1-kserve-545b7fb6bc-mk49g" podUID="3ba8dc17-1b1b-4811-9dd6-595d75c6ca04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:52.729303 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:52.726370    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6665b06d-4884-4946-947c-cbbd693d58c9-metrics-certs" pod "6665b06d-4884-4946-947c-cbbd693d58c9" (UID: "6665b06d-4884-4946-947c-cbbd693d58c9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:29:52.773171 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:52.757352    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6665b06d-4884-4946-947c-cbbd693d58c9-maas-api-tls" pod "6665b06d-4884-4946-947c-cbbd693d58c9" (UID: "6665b06d-4884-4946-947c-cbbd693d58c9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:29:52.773171 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:52.770030    4431 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6665b06d-4884-4946-947c-cbbd693d58c9-metrics-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:29:52.773171 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:52.770064    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djhdr\" (UniqueName: \"kubernetes.io/projected/6665b06d-4884-4946-947c-cbbd693d58c9-kube-api-access-djhdr\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:29:52.773171 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:52.770080    4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6665b06d-4884-4946-947c-cbbd693d58c9-maas-api-tls\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:29:52.813423 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:52.813373    4431 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5187e1/distinct-w3-5187e1-kserve-545b7fb6bc-mk49g" podUID="3ba8dc17-1b1b-4811-9dd6-595d75c6ca04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:53.105304 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:53.073046    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq"]
Nov 03 22:29:53.105304 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:53.074349    4431 cpu_manager.go:414] "containerMap: removing container" podUID="6665b06d-4884-4946-947c-cbbd693d58c9" containerName="maas-api"
Nov 03 22:29:53.105304 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:53.074388    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6665b06d-4884-4946-947c-cbbd693d58c9" containerName="maas-api"
Nov 03 22:29:53.105304 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:53.074673    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="6665b06d-4884-4946-947c-cbbd693d58c9" containerName="maas-api"
Nov 03 22:29:53.105304 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:53.097652    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq"
Nov 03 22:29:53.125240 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:53.120609    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-d05919ef-openshift-default-dockercfg-5m4h7\""
Nov 03 22:29:53.218237 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:53.217761    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc93ee10-7052-448e-999d-e2f072f4ada1-istio-data\") pod \"e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq\" (UID: \"cc93ee10-7052-448e-999d-e2f072f4ada1\") " pod="openshift-ingress/e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq"
Nov 03 22:29:53.218237 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:53.217912    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc93ee10-7052-448e-999d-e2f072f4ada1-istio-token\") pod \"e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq\" (UID: \"cc93ee10-7052-448e-999d-e2f072f4ada1\") " pod="openshift-ingress/e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq"
Nov 03 22:29:53.218237 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:53.218161    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc93ee10-7052-448e-999d-e2f072f4ada1-istiod-ca-cert\") pod \"e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq\" (UID: \"cc93ee10-7052-448e-999d-e2f072f4ada1\") " pod="openshift-ingress/e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq"
Nov 03 22:29:53.218237 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:53.218202    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhhll\" (UniqueName: \"kubernetes.io/projected/cc93ee10-7052-448e-999d-e2f072f4ada1-kube-api-access-xhhll\") pod \"e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq\" (UID: \"cc93ee10-7052-448e-999d-e2f072f4ada1\") " pod="openshift-ingress/e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq"
Nov 03 22:29:53.218581 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:53.218255    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc93ee10-7052-448e-999d-e2f072f4ada1-istio-envoy\") pod \"e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq\" (UID: \"cc93ee10-7052-448e-999d-e2f072f4ada1\") " pod="openshift-ingress/e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq"
Nov 03 22:29:53.218581 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:53.218322    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc93ee10-7052-448e-999d-e2f072f4ada1-workload-socket\") pod \"e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq\" (UID: \"cc93ee10-7052-448e-999d-e2f072f4ada1\") " pod="openshift-ingress/e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq"
Nov 03 22:29:53.218581 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:53.218383    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc93ee10-7052-448e-999d-e2f072f4ada1-workload-certs\") pod \"e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq\" (UID: \"cc93ee10-7052-448e-999d-e2f072f4ada1\") " pod="openshift-ingress/e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq"
Nov 03 22:29:53.218581 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:53.218413    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc93ee10-7052-448e-999d-e2f072f4ada1-istio-podinfo\") pod \"e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq\" (UID: \"cc93ee10-7052-448e-999d-e2f072f4ada1\") " pod="openshift-ingress/e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq"
Nov 03 22:29:53.218581 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:53.218498    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc93ee10-7052-448e-999d-e2f072f4ada1-credential-socket\") pod \"e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq\" (UID: \"cc93ee10-7052-448e-999d-e2f072f4ada1\") " pod="openshift-ingress/e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq"
Nov 03 22:29:53.271483 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:53.271442    4431 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-553753/distinct-2-w2-553753-kserve-5bf8568844-7q4l2" podUID="a1894b80-164c-4ce1-94f3-1e5011d9759e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": context deadline exceeded"
Nov 03 22:29:53.326239 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:53.324517    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:29:53.326239 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:53.325560    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc93ee10-7052-448e-999d-e2f072f4ada1-istiod-ca-cert\") pod \"e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq\" (UID: \"cc93ee10-7052-448e-999d-e2f072f4ada1\") " pod="openshift-ingress/e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq"
Nov 03 22:29:53.326239 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:53.325609    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhhll\" (UniqueName: \"kubernetes.io/projected/cc93ee10-7052-448e-999d-e2f072f4ada1-kube-api-access-xhhll\") pod \"e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq\" (UID: \"cc93ee10-7052-448e-999d-e2f072f4ada1\") " pod="openshift-ingress/e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq"
Nov 03 22:29:53.326239 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:53.325647    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc93ee10-7052-448e-999d-e2f072f4ada1-istio-envoy\") pod \"e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq\" (UID: \"cc93ee10-7052-448e-999d-e2f072f4ada1\") " pod="openshift-ingress/e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq"
Nov 03 22:29:53.326239 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:53.325708    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc93ee10-7052-448e-999d-e2f072f4ada1-workload-socket\") pod \"e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq\" (UID: \"cc93ee10-7052-448e-999d-e2f072f4ada1\") " pod="openshift-ingress/e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq"
Nov 03 22:29:53.326239 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:53.325768    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc93ee10-7052-448e-999d-e2f072f4ada1-workload-certs\") pod \"e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq\" (UID: \"cc93ee10-7052-448e-999d-e2f072f4ada1\") " pod="openshift-ingress/e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq"
Nov 03 22:29:53.326239 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:53.325801    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc93ee10-7052-448e-999d-e2f072f4ada1-istio-podinfo\") pod \"e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq\" (UID: \"cc93ee10-7052-448e-999d-e2f072f4ada1\") " pod="openshift-ingress/e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq"
Nov 03 22:29:53.326239 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:53.325878    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc93ee10-7052-448e-999d-e2f072f4ada1-credential-socket\") pod \"e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq\" (UID: \"cc93ee10-7052-448e-999d-e2f072f4ada1\") " pod="openshift-ingress/e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq"
Nov 03 22:29:53.326239 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:53.325918    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc93ee10-7052-448e-999d-e2f072f4ada1-istio-data\") pod \"e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq\" (UID: \"cc93ee10-7052-448e-999d-e2f072f4ada1\") " pod="openshift-ingress/e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq"
Nov 03 22:29:53.326239 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:53.326034    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc93ee10-7052-448e-999d-e2f072f4ada1-istio-token\") pod \"e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq\" (UID: \"cc93ee10-7052-448e-999d-e2f072f4ada1\") " pod="openshift-ingress/e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq"
Nov 03 22:29:53.332306 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:53.330332    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc93ee10-7052-448e-999d-e2f072f4ada1-istiod-ca-cert\") pod \"e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq\" (UID: \"cc93ee10-7052-448e-999d-e2f072f4ada1\") " pod="openshift-ingress/e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq"
Nov 03 22:29:53.335768 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:53.335703    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc93ee10-7052-448e-999d-e2f072f4ada1-workload-socket\") pod \"e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq\" (UID: \"cc93ee10-7052-448e-999d-e2f072f4ada1\") " pod="openshift-ingress/e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq"
Nov 03 22:29:53.336255 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:53.336202    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc93ee10-7052-448e-999d-e2f072f4ada1-credential-socket\") pod \"e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq\" (UID: \"cc93ee10-7052-448e-999d-e2f072f4ada1\") " pod="openshift-ingress/e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq"
Nov 03 22:29:53.363512 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:53.346819    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc93ee10-7052-448e-999d-e2f072f4ada1-istio-podinfo\") pod \"e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq\" (UID: \"cc93ee10-7052-448e-999d-e2f072f4ada1\") " pod="openshift-ingress/e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq"
Nov 03 22:29:53.363634 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:53.347091    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc93ee10-7052-448e-999d-e2f072f4ada1-workload-certs\") pod \"e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq\" (UID: \"cc93ee10-7052-448e-999d-e2f072f4ada1\") " pod="openshift-ingress/e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq"
Nov 03 22:29:53.363720 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:53.351632    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc93ee10-7052-448e-999d-e2f072f4ada1-istio-data\") pod \"e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq\" (UID: \"cc93ee10-7052-448e-999d-e2f072f4ada1\") " pod="openshift-ingress/e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq"
Nov 03 22:29:53.392256 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:53.382986    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhhll\" (UniqueName: \"kubernetes.io/projected/cc93ee10-7052-448e-999d-e2f072f4ada1-kube-api-access-xhhll\") pod \"e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq\" (UID: \"cc93ee10-7052-448e-999d-e2f072f4ada1\") " pod="openshift-ingress/e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq"
Nov 03 22:29:53.410584 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:53.400751    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc93ee10-7052-448e-999d-e2f072f4ada1-istio-token\") pod \"e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq\" (UID: \"cc93ee10-7052-448e-999d-e2f072f4ada1\") " pod="openshift-ingress/e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq"
Nov 03 22:29:53.435556 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:53.434651    4431 patch_prober.go:29] interesting pod/e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 03 22:29:53.435556 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:53.434713    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk" podUID="b92034e4-f537-425b-b28b-1cd528b700af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 03 22:29:53.476886 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:53.474719    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc93ee10-7052-448e-999d-e2f072f4ada1-istio-envoy\") pod \"e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq\" (UID: \"cc93ee10-7052-448e-999d-e2f072f4ada1\") " pod="openshift-ingress/e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq"
Nov 03 22:29:53.490524 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:53.478260    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b7b10505-7645b7c844-knzm7"]
Nov 03 22:29:53.490524 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:53.489626    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-50d738/distinct-w4-50d738-kserve-668fdf8b86-5pp9p" podUID="ce50af62-9cb6-4379-99c7-b40630538f4f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 03 22:29:53.496335 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:53.496118    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq"
Nov 03 22:29:53.633288 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:53.629336    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b7b10505-7645b7c844-knzm7"]
Nov 03 22:29:53.679237 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:53.678341    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq"]
Nov 03 22:29:53.933299 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:53.908804    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:29:54.257288 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:54.253991    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6665b06d-4884-4946-947c-cbbd693d58c9" path="/var/lib/kubelet/pods/6665b06d-4884-4946-947c-cbbd693d58c9/volumes"
Nov 03 22:29:54.408112 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:54.407329    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-50d738/premium-w4-50d738-kserve-59ccd7f995-xbfq2" podUID="fa0a680b-8f0d-4b19-8217-82282764511a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:54.955432 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:54.955355    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/simulator-w0-88a4d9-kserve-785f84888f-sjbzp" podUID="18bd460b-a0f1-415c-bcb4-41c9f70d2926" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded"
Nov 03 22:29:55.415259 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:55.411297    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-50d738/premium-w4-50d738-kserve-59ccd7f995-xbfq2" podUID="fa0a680b-8f0d-4b19-8217-82282764511a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:55.415259 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:55.414060    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-50d738/premium-w4-50d738-kserve-59ccd7f995-xbfq2"
Nov 03 22:29:55.416803 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:55.416761    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5187e1/premium-w3-5187e1-kserve-56bf8788d4-pp8st" podUID="a70ecfe9-020a-4331-b67c-aaaa46ed1eb1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:55.426267 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:55.422301    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-w6-2d619f-kserve-74dbc56695-tnvtb" podUID="90310d72-c818-4fdd-b4e7-b51f633caa05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:55.461285 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:55.456171    4431 patch_prober.go:29] interesting pod/e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 22:29:55.461285 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:55.456268    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk" podUID="b92034e4-f537-425b-b28b-1cd528b700af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:55.465243 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:55.464970    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-50d738/distinct-w4-50d738-kserve-668fdf8b86-5pp9p" podUID="ce50af62-9cb6-4379-99c7-b40630538f4f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:55.480986 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:55.480941    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-553753/distinct-2-w2-553753-kserve-5bf8568844-7q4l2" podUID="a1894b80-164c-4ce1-94f3-1e5011d9759e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:55.704020 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:55.703988    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-50d738/premium-w4-50d738-kserve-59ccd7f995-xbfq2"
Nov 03 22:29:55.798241 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:55.794686    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-50d738/distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r" podUID="e45b1572-6519-4c71-884a-115eeddd86cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:55.822276 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:55.818299    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5187e1/distinct-w3-5187e1-kserve-545b7fb6bc-mk49g" podUID="3ba8dc17-1b1b-4811-9dd6-595d75c6ca04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:55.847562 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:55.847517    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-553753/simulator-w2-553753-kserve-64d77d4dfc-4rlbn" podUID="a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:56.526167 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:56.526116    4431 patch_prober.go:29] interesting pod/e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 22:29:56.642098 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:56.615377    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk" podUID="b92034e4-f537-425b-b28b-1cd528b700af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:56.875657 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:56.875555    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-553753/simulator-w2-553753-kserve-64d77d4dfc-4rlbn" podUID="a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:56.876622 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:56.876592    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-553753/simulator-w2-553753-kserve-64d77d4dfc-4rlbn"
Nov 03 22:29:57.129300 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:57.126300    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:29:57.289240 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:57.238507    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-50d738/distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r" podUID="e45b1572-6519-4c71-884a-115eeddd86cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:57.289240 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:57.241008    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5187e1/distinct-w3-5187e1-kserve-545b7fb6bc-mk49g" podUID="3ba8dc17-1b1b-4811-9dd6-595d75c6ca04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:57.289240 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:57.241795    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-50d738/distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r"
Nov 03 22:29:57.289240 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:57.241883    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5187e1/distinct-w3-5187e1-kserve-545b7fb6bc-mk49g"
Nov 03 22:29:57.354249 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:57.335971    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:29:57.354249 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:57.336363    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:29:57.354249 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:57.336638    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:29:57.354249 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:57.337748    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5187e1/premium-w3-5187e1-kserve-56bf8788d4-pp8st" podUID="a70ecfe9-020a-4331-b67c-aaaa46ed1eb1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:57.413688 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:57.413642    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-w6-2d619f-kserve-74dbc56695-tnvtb" podUID="90310d72-c818-4fdd-b4e7-b51f633caa05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:57.553241 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:57.552726    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-553753/distinct-2-w2-553753-kserve-5bf8568844-7q4l2" podUID="a1894b80-164c-4ce1-94f3-1e5011d9759e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:57.636009 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:57.625868    4431 patch_prober.go:29] interesting pod/e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 22:29:57.636009 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:57.625936    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk" podUID="b92034e4-f537-425b-b28b-1cd528b700af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:57.668412 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:57.668282    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/premium-w0-88a4d9-kserve-5559d75b7b-vqllq" podUID="05206737-4ca3-41fc-b35a-37c625cc182c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:57.754394 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:57.754345    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-50d738/premium-w4-50d738-kserve-59ccd7f995-xbfq2" podUID="fa0a680b-8f0d-4b19-8217-82282764511a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 03 22:29:57.930773 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:57.930666    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-553753/simulator-w2-553753-kserve-64d77d4dfc-4rlbn" podUID="a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Nov 03 22:29:58.120241 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:57.968956    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:29:58.122900 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:58.028565    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq"]
Nov 03 22:29:58.166541 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:58.166178    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5187e1/distinct-w3-5187e1-kserve-545b7fb6bc-mk49g"
Nov 03 22:29:58.361140 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:58.361042    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-50d738/distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r" podUID="e45b1572-6519-4c71-884a-115eeddd86cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Nov 03 22:29:58.480902 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:58.480860    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq" event={"ID":"cc93ee10-7052-448e-999d-e2f072f4ada1","Type":"ContainerStarted","Data":"5db0b34023226417971f36e2fad84a951dbddd51c02d11f5e84b8abb30657de9"}
Nov 03 22:29:58.489807 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:58.489779    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:29:58.667253 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:58.666590    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:29:58.710315 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:58.690601    4431 patch_prober.go:29] interesting pod/e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 22:29:58.710315 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:58.690686    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk" podUID="b92034e4-f537-425b-b28b-1cd528b700af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:58.929568 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:58.929435    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/simulator-w0-88a4d9-kserve-785f84888f-sjbzp" podUID="18bd460b-a0f1-415c-bcb4-41c9f70d2926" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:58.953612 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:58.935493    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-553753/simulator-w2-553753-kserve-64d77d4dfc-4rlbn" podUID="a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:59.077411 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:59.077364    4431 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-50d738/distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r" podUID="e45b1572-6519-4c71-884a-115eeddd86cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:59.124610 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:59.124382    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:29:59.194194 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:59.189919    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5187e1/distinct-w3-5187e1-kserve-545b7fb6bc-mk49g" podUID="3ba8dc17-1b1b-4811-9dd6-595d75c6ca04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Nov 03 22:29:59.301489 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:59.289995    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:29:59.464508 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:59.459748    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk"
Nov 03 22:29:59.481428 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:59.481382    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-50d738/distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r" podUID="e45b1572-6519-4c71-884a-115eeddd86cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:59.482612 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:59.482577    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-553753/distinct-2-w2-553753-kserve-5bf8568844-7q4l2" podUID="a1894b80-164c-4ce1-94f3-1e5011d9759e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Nov 03 22:29:59.546167 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:59.546084    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq" event={"ID":"cc93ee10-7052-448e-999d-e2f072f4ada1","Type":"ContainerStarted","Data":"1df403008efcf9dab2d96cc57805a188dbe9795cf901647eca02d31d3f1fe498"}
Nov 03 22:29:59.645208 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:59.645141    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq" podStartSLOduration=6.645122452 podStartE2EDuration="6.645122452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:29:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:29:59.643057589 +0000 UTC m=+1023.668913448" watchObservedRunningTime="2026-11-03 22:29:59.645122452 +0000 UTC m=+1023.670978313"
Nov 03 22:29:59.755577 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:59.754779    4431 patch_prober.go:29] interesting pod/e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 22:29:59.755577 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:59.754848    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk" podUID="b92034e4-f537-425b-b28b-1cd528b700af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:29:59.775933 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:29:59.775897    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:30:00.056297 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:00.032329    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-553753/simulator-w2-553753-kserve-64d77d4dfc-4rlbn" podUID="a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:30:00.483575 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:00.483380    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-553753/distinct-2-w2-553753-kserve-5bf8568844-7q4l2" podUID="a1894b80-164c-4ce1-94f3-1e5011d9759e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:30:00.485074 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:00.485038    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-50d738/distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r" podUID="e45b1572-6519-4c71-884a-115eeddd86cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Nov 03 22:30:00.493237 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:00.485927    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-553753/distinct-2-w2-553753-kserve-5bf8568844-7q4l2"
Nov 03 22:30:00.523287 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:00.505287    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq"
Nov 03 22:30:00.579345 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:00.556900    4431 patch_prober.go:29] interesting pod/e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 03 22:30:00.579345 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:00.556959    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq" podUID="cc93ee10-7052-448e-999d-e2f072f4ada1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 03 22:30:00.706889 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:00.701492    4431 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5187e1/premium-w3-5187e1-kserve-56bf8788d4-pp8st" podUID="a70ecfe9-020a-4331-b67c-aaaa46ed1eb1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:30:00.956456 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:00.956412    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/simulator-w0-88a4d9-kserve-785f84888f-sjbzp" podUID="18bd460b-a0f1-415c-bcb4-41c9f70d2926" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded"
Nov 03 22:30:01.026261 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:01.013358    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/premium-w0-88a4d9-kserve-5559d75b7b-vqllq" podUID="05206737-4ca3-41fc-b35a-37c625cc182c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 03 22:30:01.047089 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:01.047056    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:30:01.047647 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:01.047623    4431 patch_prober.go:29] interesting pod/e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 22:30:01.047820 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:01.047799    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk" podUID="b92034e4-f537-425b-b28b-1cd528b700af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:30:01.077243 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:01.076863    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-553753/simulator-w2-553753-kserve-64d77d4dfc-4rlbn" podUID="a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:30:01.169378 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:01.154167    4431 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-553753/simulator-w2-553753-kserve-64d77d4dfc-4rlbn" podUID="a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": context deadline exceeded"
Nov 03 22:30:01.193491 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:01.192357    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5187e1/distinct-w3-5187e1-kserve-545b7fb6bc-mk49g" podUID="3ba8dc17-1b1b-4811-9dd6-595d75c6ca04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Nov 03 22:30:01.360291 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:01.358296    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5187e1/premium-w3-5187e1-kserve-56bf8788d4-pp8st" podUID="a70ecfe9-020a-4331-b67c-aaaa46ed1eb1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:30:01.369236 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:01.366092    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-50d738/distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r"
Nov 03 22:30:01.420364 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:01.420334    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-553753/simulator-w2-553753-kserve-64d77d4dfc-4rlbn"
Nov 03 22:30:01.502527 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:01.502465    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-50d738/distinct-w4-50d738-kserve-668fdf8b86-5pp9p" podUID="ce50af62-9cb6-4379-99c7-b40630538f4f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:30:01.513037 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:01.512995    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-553753/distinct-2-w2-553753-kserve-5bf8568844-7q4l2"
Nov 03 22:30:01.522271 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:01.522243    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk"
Nov 03 22:30:01.526040 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:30:01.525257    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:30:01.555661 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:01.554795    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq"
Nov 03 22:30:01.610278 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:01.585184    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk"
Nov 03 22:30:01.907196 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:01.907162    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:30:02.064901 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:02.064871    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq"
Nov 03 22:30:02.066594 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:02.066570    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq"
Nov 03 22:30:02.124263 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:02.120960    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:30:03.788575 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:03.788539    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:30:05.415551 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:05.415514    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:30:06.406492 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:06.406461    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:30:06.514053 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:06.514021    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d05919ef-5fbd9c9b67-rb26m"]
Nov 03 22:30:06.522282 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:06.522256    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d05919ef-5fbd9c9b67-rb26m"
Nov 03 22:30:06.537297 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:06.537267    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d05919ef-5fbd9c9b67-rb26m"]
Nov 03 22:30:06.544787 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:06.544751    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-d05919ef\""
Nov 03 22:30:06.549635 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:06.549605    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-d05919ef-dockercfg-vqq24\""
Nov 03 22:30:06.576097 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:06.576071    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:30:06.633991 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:06.633947    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d05919ef-98cb9cd48-4lr79"]
Nov 03 22:30:06.680850 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:06.678126    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t27c5\" (UniqueName: \"kubernetes.io/projected/14a18e02-91fb-4254-b687-644447f65f01-kube-api-access-t27c5\") pod \"payload-pre-processing-e2e-ait-d05919ef-5fbd9c9b67-rb26m\" (UID: \"14a18e02-91fb-4254-b687-644447f65f01\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d05919ef-5fbd9c9b67-rb26m"
Nov 03 22:30:06.685059 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:06.685025    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14a18e02-91fb-4254-b687-644447f65f01-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d05919ef-5fbd9c9b67-rb26m\" (UID: \"14a18e02-91fb-4254-b687-644447f65f01\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d05919ef-5fbd9c9b67-rb26m"
Nov 03 22:30:06.709974 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:06.709828    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d05919ef-98cb9cd48-4lr79"]
Nov 03 22:30:06.711418 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:06.709966    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d05919ef-98cb9cd48-4lr79"
Nov 03 22:30:06.787843 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:06.787801    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t27c5\" (UniqueName: \"kubernetes.io/projected/14a18e02-91fb-4254-b687-644447f65f01-kube-api-access-t27c5\") pod \"payload-pre-processing-e2e-ait-d05919ef-5fbd9c9b67-rb26m\" (UID: \"14a18e02-91fb-4254-b687-644447f65f01\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d05919ef-5fbd9c9b67-rb26m"
Nov 03 22:30:06.788067 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:06.787890    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2lzd\" (UniqueName: \"kubernetes.io/projected/dc03599b-31be-47bc-a0e4-eeb8ec81e3cd-kube-api-access-f2lzd\") pod \"payload-processing-e2e-ait-d05919ef-98cb9cd48-4lr79\" (UID: \"dc03599b-31be-47bc-a0e4-eeb8ec81e3cd\") " pod="openshift-ingress/payload-processing-e2e-ait-d05919ef-98cb9cd48-4lr79"
Nov 03 22:30:06.788067 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:06.787983    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc03599b-31be-47bc-a0e4-eeb8ec81e3cd-plugins-config-volume\") pod \"payload-processing-e2e-ait-d05919ef-98cb9cd48-4lr79\" (UID: \"dc03599b-31be-47bc-a0e4-eeb8ec81e3cd\") " pod="openshift-ingress/payload-processing-e2e-ait-d05919ef-98cb9cd48-4lr79"
Nov 03 22:30:06.788188 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:06.788145    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14a18e02-91fb-4254-b687-644447f65f01-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d05919ef-5fbd9c9b67-rb26m\" (UID: \"14a18e02-91fb-4254-b687-644447f65f01\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d05919ef-5fbd9c9b67-rb26m"
Nov 03 22:30:06.789152 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:06.789119    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14a18e02-91fb-4254-b687-644447f65f01-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d05919ef-5fbd9c9b67-rb26m\" (UID: \"14a18e02-91fb-4254-b687-644447f65f01\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d05919ef-5fbd9c9b67-rb26m"
Nov 03 22:30:06.812512 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:06.812477    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t27c5\" (UniqueName: \"kubernetes.io/projected/14a18e02-91fb-4254-b687-644447f65f01-kube-api-access-t27c5\") pod \"payload-pre-processing-e2e-ait-d05919ef-5fbd9c9b67-rb26m\" (UID: \"14a18e02-91fb-4254-b687-644447f65f01\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d05919ef-5fbd9c9b67-rb26m"
Nov 03 22:30:06.838264 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:06.838231    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d05919ef-5fbd9c9b67-rb26m"
Nov 03 22:30:06.890906 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:06.890865    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2lzd\" (UniqueName: \"kubernetes.io/projected/dc03599b-31be-47bc-a0e4-eeb8ec81e3cd-kube-api-access-f2lzd\") pod \"payload-processing-e2e-ait-d05919ef-98cb9cd48-4lr79\" (UID: \"dc03599b-31be-47bc-a0e4-eeb8ec81e3cd\") " pod="openshift-ingress/payload-processing-e2e-ait-d05919ef-98cb9cd48-4lr79"
Nov 03 22:30:06.891093 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:06.890973    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc03599b-31be-47bc-a0e4-eeb8ec81e3cd-plugins-config-volume\") pod \"payload-processing-e2e-ait-d05919ef-98cb9cd48-4lr79\" (UID: \"dc03599b-31be-47bc-a0e4-eeb8ec81e3cd\") " pod="openshift-ingress/payload-processing-e2e-ait-d05919ef-98cb9cd48-4lr79"
Nov 03 22:30:06.892198 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:06.892173    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc03599b-31be-47bc-a0e4-eeb8ec81e3cd-plugins-config-volume\") pod \"payload-processing-e2e-ait-d05919ef-98cb9cd48-4lr79\" (UID: \"dc03599b-31be-47bc-a0e4-eeb8ec81e3cd\") " pod="openshift-ingress/payload-processing-e2e-ait-d05919ef-98cb9cd48-4lr79"
Nov 03 22:30:06.928705 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:06.928674    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2lzd\" (UniqueName: \"kubernetes.io/projected/dc03599b-31be-47bc-a0e4-eeb8ec81e3cd-kube-api-access-f2lzd\") pod \"payload-processing-e2e-ait-d05919ef-98cb9cd48-4lr79\" (UID: \"dc03599b-31be-47bc-a0e4-eeb8ec81e3cd\") " pod="openshift-ingress/payload-processing-e2e-ait-d05919ef-98cb9cd48-4lr79"
Nov 03 22:30:07.041923 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:07.041836    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d05919ef-98cb9cd48-4lr79"
Nov 03 22:30:07.105639 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:07.105504    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:30:07.175202 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:07.175166    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d05919ef-5fbd9c9b67-rb26m"]
Nov 03 22:30:07.175427 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:30:07.175400    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14a18e02_91fb_4254_b687_644447f65f01.slice/crio-07eed472a9365700173f5e8f817a02cbd3f156211a17038c2abf086a0086316a WatchSource:0}: Error finding container 07eed472a9365700173f5e8f817a02cbd3f156211a17038c2abf086a0086316a: Status 404 returned error can't find the container with id 07eed472a9365700173f5e8f817a02cbd3f156211a17038c2abf086a0086316a
Nov 03 22:30:07.346563 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:07.346294    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d05919ef-98cb9cd48-4lr79"]
Nov 03 22:30:07.347866 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:30:07.347839    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc03599b_31be_47bc_a0e4_eeb8ec81e3cd.slice/crio-99061f0993105c76667f782e6c08e5ee5f995c98ab1804309278e2b17d5191be WatchSource:0}: Error finding container 99061f0993105c76667f782e6c08e5ee5f995c98ab1804309278e2b17d5191be: Status 404 returned error can't find the container with id 99061f0993105c76667f782e6c08e5ee5f995c98ab1804309278e2b17d5191be
Nov 03 22:30:08.107855 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:08.107822    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d05919ef-98cb9cd48-4lr79" event={"ID":"dc03599b-31be-47bc-a0e4-eeb8ec81e3cd","Type":"ContainerStarted","Data":"06e2a91457f9255ff6179c3dd48abab329ec67c99e499418a098a9af2672dffa"}
Nov 03 22:30:08.108383 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:08.107870    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d05919ef-98cb9cd48-4lr79" event={"ID":"dc03599b-31be-47bc-a0e4-eeb8ec81e3cd","Type":"ContainerStarted","Data":"99061f0993105c76667f782e6c08e5ee5f995c98ab1804309278e2b17d5191be"}
Nov 03 22:30:08.108383 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:08.108025    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-d05919ef-98cb9cd48-4lr79"
Nov 03 22:30:08.109764 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:08.109731    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d05919ef-5fbd9c9b67-rb26m" event={"ID":"14a18e02-91fb-4254-b687-644447f65f01","Type":"ContainerStarted","Data":"ef635b397f62150881a2544b616520829c3e02c7f90707b58c45518ccf330aee"}
Nov 03 22:30:08.109882 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:08.109788    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d05919ef-5fbd9c9b67-rb26m" event={"ID":"14a18e02-91fb-4254-b687-644447f65f01","Type":"ContainerStarted","Data":"07eed472a9365700173f5e8f817a02cbd3f156211a17038c2abf086a0086316a"}
Nov 03 22:30:08.109882 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:08.109816    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-d05919ef-5fbd9c9b67-rb26m"
Nov 03 22:30:08.150839 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:08.150790    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-d05919ef-98cb9cd48-4lr79" podStartSLOduration=2.150779295 podStartE2EDuration="2.150779295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:30:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:30:08.14743566 +0000 UTC m=+1032.173291523" watchObservedRunningTime="2026-11-03 22:30:08.150779295 +0000 UTC m=+1032.176635157"
Nov 03 22:30:08.192172 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:08.192123    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-d05919ef-5fbd9c9b67-rb26m" podStartSLOduration=2.192109721 podStartE2EDuration="2.192109721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:30:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:30:08.189757423 +0000 UTC m=+1032.215613282" watchObservedRunningTime="2026-11-03 22:30:08.192109721 +0000 UTC m=+1032.217965584"
Nov 03 22:30:08.858582 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:08.858539    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:30:09.014739 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:09.014689    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:30:09.245412 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:09.245377    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:30:10.722830 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:10.722747    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:30:12.280492 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:12.280455    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:30:12.425248 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:12.425196    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:30:12.602344 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:12.600092    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:30:14.090789 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:14.090749    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:30:14.181314 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:14.181279    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5"]
Nov 03 22:30:14.185967 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:14.185939    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5"
Nov 03 22:30:14.206564 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:14.206502    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h969q\" (UniqueName: \"kubernetes.io/projected/b37263ad-40ef-4f70-a5fd-a2a1f14a3534-kube-api-access-h969q\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5\" (UID: \"b37263ad-40ef-4f70-a5fd-a2a1f14a3534\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5"
Nov 03 22:30:14.206709 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:14.206576    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b37263ad-40ef-4f70-a5fd-a2a1f14a3534-workload-certs\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5\" (UID: \"b37263ad-40ef-4f70-a5fd-a2a1f14a3534\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5"
Nov 03 22:30:14.206709 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:14.206633    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b37263ad-40ef-4f70-a5fd-a2a1f14a3534-workload-socket\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5\" (UID: \"b37263ad-40ef-4f70-a5fd-a2a1f14a3534\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5"
Nov 03 22:30:14.206789 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:14.206716    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b37263ad-40ef-4f70-a5fd-a2a1f14a3534-istio-envoy\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5\" (UID: \"b37263ad-40ef-4f70-a5fd-a2a1f14a3534\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5"
Nov 03 22:30:14.206789 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:14.206751    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b37263ad-40ef-4f70-a5fd-a2a1f14a3534-istio-data\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5\" (UID: \"b37263ad-40ef-4f70-a5fd-a2a1f14a3534\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5"
Nov 03 22:30:14.206789 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:14.206780    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b37263ad-40ef-4f70-a5fd-a2a1f14a3534-istio-podinfo\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5\" (UID: \"b37263ad-40ef-4f70-a5fd-a2a1f14a3534\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5"
Nov 03 22:30:14.206893 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:14.206809    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b37263ad-40ef-4f70-a5fd-a2a1f14a3534-istio-token\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5\" (UID: \"b37263ad-40ef-4f70-a5fd-a2a1f14a3534\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5"
Nov 03 22:30:14.206893 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:14.206824    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b37263ad-40ef-4f70-a5fd-a2a1f14a3534-istiod-ca-cert\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5\" (UID: \"b37263ad-40ef-4f70-a5fd-a2a1f14a3534\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5"
Nov 03 22:30:14.206893 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:14.206881    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b37263ad-40ef-4f70-a5fd-a2a1f14a3534-credential-socket\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5\" (UID: \"b37263ad-40ef-4f70-a5fd-a2a1f14a3534\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5"
Nov 03 22:30:14.226691 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:14.226658    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5"]
Nov 03 22:30:14.250435 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:14.250406    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-cbe6e297-848c8d947f-6rcr7"]
Nov 03 22:30:14.258513 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:14.258488    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-cbe6e297-848c8d947f-6rcr7"
Nov 03 22:30:14.277013 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:14.276983    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-cbe6e297-848c8d947f-6rcr7"]
Nov 03 22:30:14.278405 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:14.278331    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-cbe6e297\""
Nov 03 22:30:14.278608 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:14.278577    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-cbe6e297-dockercfg-4w6tx\""
Nov 03 22:30:14.283674 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:14.283644    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:30:14.310836 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:14.310804    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b37263ad-40ef-4f70-a5fd-a2a1f14a3534-credential-socket\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5\" (UID: \"b37263ad-40ef-4f70-a5fd-a2a1f14a3534\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5"
Nov 03 22:30:14.311040 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:14.310911    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p8z2\" (UniqueName: \"kubernetes.io/projected/48ef2535-a763-4dd1-9a0d-787464c05720-kube-api-access-5p8z2\") pod \"payload-pre-processing-e2e-derive-cbe6e297-848c8d947f-6rcr7\" (UID: \"48ef2535-a763-4dd1-9a0d-787464c05720\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-cbe6e297-848c8d947f-6rcr7"
Nov 03 22:30:14.311040 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:14.311027    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h969q\" (UniqueName: \"kubernetes.io/projected/b37263ad-40ef-4f70-a5fd-a2a1f14a3534-kube-api-access-h969q\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5\" (UID: \"b37263ad-40ef-4f70-a5fd-a2a1f14a3534\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5"
Nov 03 22:30:14.311168 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:14.311106    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b37263ad-40ef-4f70-a5fd-a2a1f14a3534-workload-certs\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5\" (UID: \"b37263ad-40ef-4f70-a5fd-a2a1f14a3534\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5"
Nov 03 22:30:14.311168 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:14.311138    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b37263ad-40ef-4f70-a5fd-a2a1f14a3534-workload-socket\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5\" (UID: \"b37263ad-40ef-4f70-a5fd-a2a1f14a3534\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5"
Nov 03 22:30:14.312306 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:14.311304    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48ef2535-a763-4dd1-9a0d-787464c05720-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-cbe6e297-848c8d947f-6rcr7\" (UID: \"48ef2535-a763-4dd1-9a0d-787464c05720\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-cbe6e297-848c8d947f-6rcr7"
Nov 03 22:30:14.312306 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:14.311964    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b37263ad-40ef-4f70-a5fd-a2a1f14a3534-credential-socket\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5\" (UID: \"b37263ad-40ef-4f70-a5fd-a2a1f14a3534\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5"
Nov 03 22:30:14.312306 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:14.312267    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b37263ad-40ef-4f70-a5fd-a2a1f14a3534-workload-socket\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5\" (UID: \"b37263ad-40ef-4f70-a5fd-a2a1f14a3534\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5"
Nov 03 22:30:14.312530 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:14.312330    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b37263ad-40ef-4f70-a5fd-a2a1f14a3534-workload-certs\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5\" (UID: \"b37263ad-40ef-4f70-a5fd-a2a1f14a3534\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5"
Nov 03 22:30:14.312530 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:14.312454    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b37263ad-40ef-4f70-a5fd-a2a1f14a3534-istio-envoy\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5\" (UID: \"b37263ad-40ef-4f70-a5fd-a2a1f14a3534\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5"
Nov 03 22:30:14.312651 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:14.312599    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b37263ad-40ef-4f70-a5fd-a2a1f14a3534-istio-data\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5\" (UID: \"b37263ad-40ef-4f70-a5fd-a2a1f14a3534\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5"
Nov 03 22:30:14.312725 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:14.312706    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b37263ad-40ef-4f70-a5fd-a2a1f14a3534-istio-podinfo\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5\" (UID: \"b37263ad-40ef-4f70-a5fd-a2a1f14a3534\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5"
Nov 03 22:30:14.313254 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:14.313129    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b37263ad-40ef-4f70-a5fd-a2a1f14a3534-istio-token\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5\" (UID: \"b37263ad-40ef-4f70-a5fd-a2a1f14a3534\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5"
Nov 03 22:30:14.315257 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:14.313867    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b37263ad-40ef-4f70-a5fd-a2a1f14a3534-istio-data\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5\" (UID: \"b37263ad-40ef-4f70-a5fd-a2a1f14a3534\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5"
Nov 03 22:30:14.315257 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:14.314286    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b37263ad-40ef-4f70-a5fd-a2a1f14a3534-istiod-ca-cert\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5\" (UID: \"b37263ad-40ef-4f70-a5fd-a2a1f14a3534\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5"
Nov 03 22:30:14.315257 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:14.314952    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b37263ad-40ef-4f70-a5fd-a2a1f14a3534-istiod-ca-cert\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5\" (UID: \"b37263ad-40ef-4f70-a5fd-a2a1f14a3534\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5"
Nov 03 22:30:14.317432 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:14.317406    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b37263ad-40ef-4f70-a5fd-a2a1f14a3534-istio-podinfo\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5\" (UID: \"b37263ad-40ef-4f70-a5fd-a2a1f14a3534\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5"
Nov 03 22:30:14.317866 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:14.317840    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b37263ad-40ef-4f70-a5fd-a2a1f14a3534-istio-envoy\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5\" (UID: \"b37263ad-40ef-4f70-a5fd-a2a1f14a3534\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5"
Nov 03 22:30:14.332592 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:14.332552    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-cbe6e297-545ffc4c86-qnkpg"]
Nov 03 22:30:14.335693 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:14.335649    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h969q\" (UniqueName: \"kubernetes.io/projected/b37263ad-40ef-4f70-a5fd-a2a1f14a3534-kube-api-access-h969q\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5\" (UID: \"b37263ad-40ef-4f70-a5fd-a2a1f14a3534\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5"
Nov 03 22:30:14.337950 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:14.337931    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b37263ad-40ef-4f70-a5fd-a2a1f14a3534-istio-token\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5\" (UID: \"b37263ad-40ef-4f70-a5fd-a2a1f14a3534\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5"
Nov 03 22:30:14.339053 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:14.339037    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-cbe6e297-545ffc4c86-qnkpg"
Nov 03 22:30:14.417285 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:14.416567    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p8z2\" (UniqueName: \"kubernetes.io/projected/48ef2535-a763-4dd1-9a0d-787464c05720-kube-api-access-5p8z2\") pod \"payload-pre-processing-e2e-derive-cbe6e297-848c8d947f-6rcr7\" (UID: \"48ef2535-a763-4dd1-9a0d-787464c05720\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-cbe6e297-848c8d947f-6rcr7"
Nov 03 22:30:14.417285 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:14.416766    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10ce4d4e-6f02-4163-9723-df0c2a18f7bf-plugins-config-volume\") pod \"payload-processing-e2e-derive-cbe6e297-545ffc4c86-qnkpg\" (UID: \"10ce4d4e-6f02-4163-9723-df0c2a18f7bf\") " pod="openshift-ingress/payload-processing-e2e-derive-cbe6e297-545ffc4c86-qnkpg"
Nov 03 22:30:14.417285 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:14.416871    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48ef2535-a763-4dd1-9a0d-787464c05720-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-cbe6e297-848c8d947f-6rcr7\" (UID: \"48ef2535-a763-4dd1-9a0d-787464c05720\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-cbe6e297-848c8d947f-6rcr7"
Nov 03 22:30:14.417285 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:14.416937    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gks79\" (UniqueName: \"kubernetes.io/projected/10ce4d4e-6f02-4163-9723-df0c2a18f7bf-kube-api-access-gks79\") pod \"payload-processing-e2e-derive-cbe6e297-545ffc4c86-qnkpg\" (UID: \"10ce4d4e-6f02-4163-9723-df0c2a18f7bf\") " pod="openshift-ingress/payload-processing-e2e-derive-cbe6e297-545ffc4c86-qnkpg"
Nov 03 22:30:14.418597 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:14.418465    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48ef2535-a763-4dd1-9a0d-787464c05720-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-cbe6e297-848c8d947f-6rcr7\" (UID: \"48ef2535-a763-4dd1-9a0d-787464c05720\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-cbe6e297-848c8d947f-6rcr7"
Nov 03 22:30:14.426950 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:14.426917    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:30:14.459475 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:14.459364    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-cbe6e297-545ffc4c86-qnkpg"]
Nov 03 22:30:14.469669 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:14.469636    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p8z2\" (UniqueName: \"kubernetes.io/projected/48ef2535-a763-4dd1-9a0d-787464c05720-kube-api-access-5p8z2\") pod \"payload-pre-processing-e2e-derive-cbe6e297-848c8d947f-6rcr7\" (UID: \"48ef2535-a763-4dd1-9a0d-787464c05720\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-cbe6e297-848c8d947f-6rcr7"
Nov 03 22:30:14.508703 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:14.508345    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5"
Nov 03 22:30:14.519286 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:14.519209    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10ce4d4e-6f02-4163-9723-df0c2a18f7bf-plugins-config-volume\") pod \"payload-processing-e2e-derive-cbe6e297-545ffc4c86-qnkpg\" (UID: \"10ce4d4e-6f02-4163-9723-df0c2a18f7bf\") " pod="openshift-ingress/payload-processing-e2e-derive-cbe6e297-545ffc4c86-qnkpg"
Nov 03 22:30:14.519418 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:14.519330    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gks79\" (UniqueName: \"kubernetes.io/projected/10ce4d4e-6f02-4163-9723-df0c2a18f7bf-kube-api-access-gks79\") pod \"payload-processing-e2e-derive-cbe6e297-545ffc4c86-qnkpg\" (UID: \"10ce4d4e-6f02-4163-9723-df0c2a18f7bf\") " pod="openshift-ingress/payload-processing-e2e-derive-cbe6e297-545ffc4c86-qnkpg"
Nov 03 22:30:14.520305 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:14.520271    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10ce4d4e-6f02-4163-9723-df0c2a18f7bf-plugins-config-volume\") pod \"payload-processing-e2e-derive-cbe6e297-545ffc4c86-qnkpg\" (UID: \"10ce4d4e-6f02-4163-9723-df0c2a18f7bf\") " pod="openshift-ingress/payload-processing-e2e-derive-cbe6e297-545ffc4c86-qnkpg"
Nov 03 22:30:14.542599 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:14.542562    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gks79\" (UniqueName: \"kubernetes.io/projected/10ce4d4e-6f02-4163-9723-df0c2a18f7bf-kube-api-access-gks79\") pod \"payload-processing-e2e-derive-cbe6e297-545ffc4c86-qnkpg\" (UID: \"10ce4d4e-6f02-4163-9723-df0c2a18f7bf\") " pod="openshift-ingress/payload-processing-e2e-derive-cbe6e297-545ffc4c86-qnkpg"
Nov 03 22:30:14.576952 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:14.576921    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-cbe6e297-848c8d947f-6rcr7"
Nov 03 22:30:14.734242 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:14.733672    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-cbe6e297-545ffc4c86-qnkpg"
Nov 03 22:30:14.853319 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:14.852858    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5"]
Nov 03 22:30:14.867207 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:14.867158    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:30:14.867346 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:14.867278    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:30:14.867346 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:14.867323    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:30:15.052808 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:15.052777    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-cbe6e297-848c8d947f-6rcr7"]
Nov 03 22:30:15.057589 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:30:15.057556    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48ef2535_a763_4dd1_9a0d_787464c05720.slice/crio-fd38a7bdd7aff42c8371d1c6dfc81050705c1d31d8a1bad34eb250ba46110f51 WatchSource:0}: Error finding container fd38a7bdd7aff42c8371d1c6dfc81050705c1d31d8a1bad34eb250ba46110f51: Status 404 returned error can't find the container with id fd38a7bdd7aff42c8371d1c6dfc81050705c1d31d8a1bad34eb250ba46110f51
Nov 03 22:30:15.167195 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:15.167156    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-cbe6e297-848c8d947f-6rcr7" event={"ID":"48ef2535-a763-4dd1-9a0d-787464c05720","Type":"ContainerStarted","Data":"e7c41627aefd379929cd1271f7f7464d68d9ed8bf8d519a1018a639270924b53"}
Nov 03 22:30:15.167783 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:15.167234    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-cbe6e297-848c8d947f-6rcr7" event={"ID":"48ef2535-a763-4dd1-9a0d-787464c05720","Type":"ContainerStarted","Data":"fd38a7bdd7aff42c8371d1c6dfc81050705c1d31d8a1bad34eb250ba46110f51"}
Nov 03 22:30:15.167783 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:15.167266    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-cbe6e297-848c8d947f-6rcr7"
Nov 03 22:30:15.169175 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:15.169151    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5" event={"ID":"b37263ad-40ef-4f70-a5fd-a2a1f14a3534","Type":"ContainerStarted","Data":"6fd3924b880e294f5fcf394f52a0a8a1cdfa65e442e5b58c253c58ca7471b272"}
Nov 03 22:30:15.169346 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:15.169198    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5" event={"ID":"b37263ad-40ef-4f70-a5fd-a2a1f14a3534","Type":"ContainerStarted","Data":"6b84c339727786d6a49433098e9171fad6fb31e544d7722856644999c0b709b1"}
Nov 03 22:30:15.174760 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:15.174727    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-cbe6e297-545ffc4c86-qnkpg"]
Nov 03 22:30:15.179485 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:30:15.179458    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10ce4d4e_6f02_4163_9723_df0c2a18f7bf.slice/crio-e6fd7bda691ba46e4ce048e862084fe789971dfbd892aa34300c30612f08228d WatchSource:0}: Error finding container e6fd7bda691ba46e4ce048e862084fe789971dfbd892aa34300c30612f08228d: Status 404 returned error can't find the container with id e6fd7bda691ba46e4ce048e862084fe789971dfbd892aa34300c30612f08228d
Nov 03 22:30:15.213786 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:15.213728    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-cbe6e297-848c8d947f-6rcr7" podStartSLOduration=1.213707236 podStartE2EDuration="1.213707236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:30:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:30:15.211094716 +0000 UTC m=+1039.236950579" watchObservedRunningTime="2026-11-03 22:30:15.213707236 +0000 UTC m=+1039.239563099"
Nov 03 22:30:15.258542 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:15.258489    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5" podStartSLOduration=1.258470457 podStartE2EDuration="1.258470457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:30:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:30:15.257439698 +0000 UTC m=+1039.283295559" watchObservedRunningTime="2026-11-03 22:30:15.258470457 +0000 UTC m=+1039.284326320"
Nov 03 22:30:15.509277 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:15.509157    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5"
Nov 03 22:30:15.511307 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:15.511270    4431 patch_prober.go:29] interesting pod/e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 03 22:30:15.511483 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:15.511325    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5" podUID="b37263ad-40ef-4f70-a5fd-a2a1f14a3534" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 03 22:30:16.179490 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:16.179442    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:30:16.191871 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:16.191827    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-cbe6e297-545ffc4c86-qnkpg" event={"ID":"10ce4d4e-6f02-4163-9723-df0c2a18f7bf","Type":"ContainerStarted","Data":"5541f8bf0c23047ce49346f83f9666822300206acb13772ef7ab8a8e9b91ca71"}
Nov 03 22:30:16.192185 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:16.192164    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-cbe6e297-545ffc4c86-qnkpg"
Nov 03 22:30:16.192317 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:16.192206    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-cbe6e297-545ffc4c86-qnkpg" event={"ID":"10ce4d4e-6f02-4163-9723-df0c2a18f7bf","Type":"ContainerStarted","Data":"e6fd7bda691ba46e4ce048e862084fe789971dfbd892aa34300c30612f08228d"}
Nov 03 22:30:16.234263 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:16.234194    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-cbe6e297-545ffc4c86-qnkpg" podStartSLOduration=2.234179794 podStartE2EDuration="2.234179794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:30:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:30:16.231300458 +0000 UTC m=+1040.257156323" watchObservedRunningTime="2026-11-03 22:30:16.234179794 +0000 UTC m=+1040.260035654"
Nov 03 22:30:16.509496 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:16.509418    4431 patch_prober.go:29] interesting pod/e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 03 22:30:16.509496 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:16.509472    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5" podUID="b37263ad-40ef-4f70-a5fd-a2a1f14a3534" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 03 22:30:17.509000 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:17.508963    4431 patch_prober.go:29] interesting pod/e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 03 22:30:17.509489 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:17.509017    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5" podUID="b37263ad-40ef-4f70-a5fd-a2a1f14a3534" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 03 22:30:17.792911 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:17.792834    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:30:18.509312 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:18.509280    4431 patch_prober.go:29] interesting pod/e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 03 22:30:18.509706 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:18.509340    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5" podUID="b37263ad-40ef-4f70-a5fd-a2a1f14a3534" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 03 22:30:18.868275 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:18.868147    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-51bfa2e1-6954676576-kr8sq"]
Nov 03 22:30:19.065703 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:19.065625    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:30:19.120720 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:19.120451    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-d05919ef-5fbd9c9b67-rb26m"
Nov 03 22:30:19.121080 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:19.121052    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-d05919ef-98cb9cd48-4lr79"
Nov 03 22:30:19.335689 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:19.335658    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:30:19.509894 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:19.509854    4431 patch_prober.go:29] interesting pod/e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 03 22:30:19.510507 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:19.510473    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5" podUID="b37263ad-40ef-4f70-a5fd-a2a1f14a3534" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 03 22:30:19.570875 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:19.570835    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:30:19.711995 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:19.711954    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:30:20.359107 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:20.359053    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:30:20.510585 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:20.510362    4431 patch_prober.go:29] interesting pod/e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 03 22:30:20.510585 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:20.510424    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5" podUID="b37263ad-40ef-4f70-a5fd-a2a1f14a3534" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 03 22:30:21.127583 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:21.127547    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:30:21.508998 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:21.508964    4431 patch_prober.go:29] interesting pod/e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 03 22:30:21.509229 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:21.509039    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5" podUID="b37263ad-40ef-4f70-a5fd-a2a1f14a3534" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 03 22:30:21.678845 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:21.678806    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:30:21.825263 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:21.824923    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:30:22.479263 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:22.479183    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:30:22.752292 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:22.751929    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5"
Nov 03 22:30:22.753672 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:22.753389    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5"
Nov 03 22:30:22.754900 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:22.754875    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5"
Nov 03 22:30:23.391822 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:23.391787    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:30:23.546971 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:23.546918    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:30:23.837721 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:23.837684    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:30:23.868974 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:23.868933    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-51bfa2e1-6954676576-kr8sq" podUID="5e679b58-cace-47c0-8673-3f9e9addea36" containerName="payload-processing" containerID="cri-o://0a09c5fb968f0b598019d4116cc86794adecae0eb5dfb9b4ee276620269c2bdc" gracePeriod=25
Nov 03 22:30:24.097968 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:24.095959    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2xmvs"]
Nov 03 22:30:24.103983 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:24.101862    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2xmvs"
Nov 03 22:30:24.134687 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:24.134652    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2xmvs"]
Nov 03 22:30:24.206561 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:24.206527    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 03 22:30:24.206713 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:24.206607    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 03 22:30:24.206713 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:24.206648    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 03 22:30:24.206859 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:24.206836    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-h854s\""
Nov 03 22:30:24.207419 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:24.207383    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv7gg\" (UniqueName: \"kubernetes.io/projected/152e2bae-faf4-4d6f-9791-01fbbe4581c3-kube-api-access-rv7gg\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2xmvs\" (UID: \"152e2bae-faf4-4d6f-9791-01fbbe4581c3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2xmvs"
Nov 03 22:30:24.207534 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:24.207467    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/152e2bae-faf4-4d6f-9791-01fbbe4581c3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2xmvs\" (UID: \"152e2bae-faf4-4d6f-9791-01fbbe4581c3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2xmvs"
Nov 03 22:30:24.207596 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:24.207548    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/152e2bae-faf4-4d6f-9791-01fbbe4581c3-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2xmvs\" (UID: \"152e2bae-faf4-4d6f-9791-01fbbe4581c3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2xmvs"
Nov 03 22:30:24.207743 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:24.207724    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/152e2bae-faf4-4d6f-9791-01fbbe4581c3-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2xmvs\" (UID: \"152e2bae-faf4-4d6f-9791-01fbbe4581c3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2xmvs"
Nov 03 22:30:24.207818 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:24.207792    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/152e2bae-faf4-4d6f-9791-01fbbe4581c3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2xmvs\" (UID: \"152e2bae-faf4-4d6f-9791-01fbbe4581c3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2xmvs"
Nov 03 22:30:24.207876 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:24.207833    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/152e2bae-faf4-4d6f-9791-01fbbe4581c3-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2xmvs\" (UID: \"152e2bae-faf4-4d6f-9791-01fbbe4581c3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2xmvs"
Nov 03 22:30:24.207876 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:24.207862    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/152e2bae-faf4-4d6f-9791-01fbbe4581c3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2xmvs\" (UID: \"152e2bae-faf4-4d6f-9791-01fbbe4581c3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2xmvs"
Nov 03 22:30:24.216369 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:24.216351    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-51bfa2e1-6954676576-kr8sq"
Nov 03 22:30:24.262172 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:24.262137    4431 generic.go:356] "Generic (PLEG): container finished" podID="5e679b58-cace-47c0-8673-3f9e9addea36" containerID="0a09c5fb968f0b598019d4116cc86794adecae0eb5dfb9b4ee276620269c2bdc" exitCode=0
Nov 03 22:30:24.262410 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:24.262234    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-51bfa2e1-6954676576-kr8sq"
Nov 03 22:30:24.262410 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:24.262252    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-51bfa2e1-6954676576-kr8sq" event={"ID":"5e679b58-cace-47c0-8673-3f9e9addea36","Type":"ContainerDied","Data":"0a09c5fb968f0b598019d4116cc86794adecae0eb5dfb9b4ee276620269c2bdc"}
Nov 03 22:30:24.262410 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:24.262311    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-51bfa2e1-6954676576-kr8sq" event={"ID":"5e679b58-cace-47c0-8673-3f9e9addea36","Type":"ContainerDied","Data":"b35f342c4be0bb6b4c96c551fe413adeb29583707532b1d8d2372ecb6e514d66"}
Nov 03 22:30:24.262410 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:24.262334    4431 scope.go:122] "RemoveContainer" containerID="0a09c5fb968f0b598019d4116cc86794adecae0eb5dfb9b4ee276620269c2bdc"
Nov 03 22:30:24.274182 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:24.274158    4431 scope.go:122] "RemoveContainer" containerID="0a09c5fb968f0b598019d4116cc86794adecae0eb5dfb9b4ee276620269c2bdc"
Nov 03 22:30:24.274582 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:30:24.274558    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a09c5fb968f0b598019d4116cc86794adecae0eb5dfb9b4ee276620269c2bdc\": container with ID starting with 0a09c5fb968f0b598019d4116cc86794adecae0eb5dfb9b4ee276620269c2bdc not found: ID does not exist" containerID="0a09c5fb968f0b598019d4116cc86794adecae0eb5dfb9b4ee276620269c2bdc"
Nov 03 22:30:24.274678 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:24.274589    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a09c5fb968f0b598019d4116cc86794adecae0eb5dfb9b4ee276620269c2bdc"} err="failed to get container status \"0a09c5fb968f0b598019d4116cc86794adecae0eb5dfb9b4ee276620269c2bdc\": rpc error: code = NotFound desc = could not find container \"0a09c5fb968f0b598019d4116cc86794adecae0eb5dfb9b4ee276620269c2bdc\": container with ID starting with 0a09c5fb968f0b598019d4116cc86794adecae0eb5dfb9b4ee276620269c2bdc not found: ID does not exist"
Nov 03 22:30:24.309165 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:24.309083    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e679b58-cace-47c0-8673-3f9e9addea36-kube-api-access-9tdfp\" (UniqueName: \"kubernetes.io/projected/5e679b58-cace-47c0-8673-3f9e9addea36-kube-api-access-9tdfp\") pod \"5e679b58-cace-47c0-8673-3f9e9addea36\" (UID: \"5e679b58-cace-47c0-8673-3f9e9addea36\") "
Nov 03 22:30:24.309412 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:24.309258    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e679b58-cace-47c0-8673-3f9e9addea36-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e679b58-cace-47c0-8673-3f9e9addea36-plugins-config-volume\") pod \"5e679b58-cace-47c0-8673-3f9e9addea36\" (UID: \"5e679b58-cace-47c0-8673-3f9e9addea36\") "
Nov 03 22:30:24.309677 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:24.309654    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/152e2bae-faf4-4d6f-9791-01fbbe4581c3-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2xmvs\" (UID: \"152e2bae-faf4-4d6f-9791-01fbbe4581c3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2xmvs"
Nov 03 22:30:24.309799 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:24.309742    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/152e2bae-faf4-4d6f-9791-01fbbe4581c3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2xmvs\" (UID: \"152e2bae-faf4-4d6f-9791-01fbbe4581c3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2xmvs"
Nov 03 22:30:24.309865 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:24.309803    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/152e2bae-faf4-4d6f-9791-01fbbe4581c3-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2xmvs\" (UID: \"152e2bae-faf4-4d6f-9791-01fbbe4581c3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2xmvs"
Nov 03 22:30:24.309865 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:24.309840    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/152e2bae-faf4-4d6f-9791-01fbbe4581c3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2xmvs\" (UID: \"152e2bae-faf4-4d6f-9791-01fbbe4581c3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2xmvs"
Nov 03 22:30:24.309985 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:24.309898    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e679b58-cace-47c0-8673-3f9e9addea36-plugins-config-volume" pod "5e679b58-cace-47c0-8673-3f9e9addea36" (UID: "5e679b58-cace-47c0-8673-3f9e9addea36"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:30:24.310064 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:24.310039    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv7gg\" (UniqueName: \"kubernetes.io/projected/152e2bae-faf4-4d6f-9791-01fbbe4581c3-kube-api-access-rv7gg\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2xmvs\" (UID: \"152e2bae-faf4-4d6f-9791-01fbbe4581c3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2xmvs"
Nov 03 22:30:24.310145 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:24.310127    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/152e2bae-faf4-4d6f-9791-01fbbe4581c3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2xmvs\" (UID: \"152e2bae-faf4-4d6f-9791-01fbbe4581c3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2xmvs"
Nov 03 22:30:24.310283 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:24.310265    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/152e2bae-faf4-4d6f-9791-01fbbe4581c3-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2xmvs\" (UID: \"152e2bae-faf4-4d6f-9791-01fbbe4581c3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2xmvs"
Nov 03 22:30:24.310599 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:24.310573    4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e679b58-cace-47c0-8673-3f9e9addea36-plugins-config-volume\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:30:24.311370 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:24.311327    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/152e2bae-faf4-4d6f-9791-01fbbe4581c3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2xmvs\" (UID: \"152e2bae-faf4-4d6f-9791-01fbbe4581c3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2xmvs"
Nov 03 22:30:24.313605 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:24.313559    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/152e2bae-faf4-4d6f-9791-01fbbe4581c3-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2xmvs\" (UID: \"152e2bae-faf4-4d6f-9791-01fbbe4581c3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2xmvs"
Nov 03 22:30:24.313755 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:24.313730    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/152e2bae-faf4-4d6f-9791-01fbbe4581c3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2xmvs\" (UID: \"152e2bae-faf4-4d6f-9791-01fbbe4581c3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2xmvs"
Nov 03 22:30:24.313810 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:24.313777    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e679b58-cace-47c0-8673-3f9e9addea36-kube-api-access-9tdfp" pod "5e679b58-cace-47c0-8673-3f9e9addea36" (UID: "5e679b58-cace-47c0-8673-3f9e9addea36"). InnerVolumeSpecName "kube-api-access-9tdfp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:30:24.314811 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:24.314789    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/152e2bae-faf4-4d6f-9791-01fbbe4581c3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2xmvs\" (UID: \"152e2bae-faf4-4d6f-9791-01fbbe4581c3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2xmvs"
Nov 03 22:30:24.319356 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:24.319332    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/152e2bae-faf4-4d6f-9791-01fbbe4581c3-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2xmvs\" (UID: \"152e2bae-faf4-4d6f-9791-01fbbe4581c3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2xmvs"
Nov 03 22:30:24.337916 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:24.337889    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv7gg\" (UniqueName: \"kubernetes.io/projected/152e2bae-faf4-4d6f-9791-01fbbe4581c3-kube-api-access-rv7gg\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2xmvs\" (UID: \"152e2bae-faf4-4d6f-9791-01fbbe4581c3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2xmvs"
Nov 03 22:30:24.340052 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:24.340024    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/152e2bae-faf4-4d6f-9791-01fbbe4581c3-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2xmvs\" (UID: \"152e2bae-faf4-4d6f-9791-01fbbe4581c3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2xmvs"
Nov 03 22:30:24.412339 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:24.412312    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tdfp\" (UniqueName: \"kubernetes.io/projected/5e679b58-cace-47c0-8673-3f9e9addea36-kube-api-access-9tdfp\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:30:24.423638 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:24.423607    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2xmvs"
Nov 03 22:30:24.615946 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:24.615881    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-51bfa2e1-6954676576-kr8sq"]
Nov 03 22:30:24.619773 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:24.619745    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-51bfa2e1-6954676576-kr8sq"]
Nov 03 22:30:24.684435 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:24.684345    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:30:24.758967 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:30:24.758925    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod152e2bae_faf4_4d6f_9791_01fbbe4581c3.slice/crio-8b085f3a1d9a577bc4a99b811dd41071da0ef3de39cbdb03f8e591bc472f7b32 WatchSource:0}: Error finding container 8b085f3a1d9a577bc4a99b811dd41071da0ef3de39cbdb03f8e591bc472f7b32: Status 404 returned error can't find the container with id 8b085f3a1d9a577bc4a99b811dd41071da0ef3de39cbdb03f8e591bc472f7b32
Nov 03 22:30:24.762271 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:24.762193    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2xmvs"]
Nov 03 22:30:24.832501 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:24.832465    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:30:25.270542 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:25.270504    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2xmvs" event={"ID":"152e2bae-faf4-4d6f-9791-01fbbe4581c3","Type":"ContainerStarted","Data":"8b085f3a1d9a577bc4a99b811dd41071da0ef3de39cbdb03f8e591bc472f7b32"}
Nov 03 22:30:25.709947 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:25.709916    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:30:26.090905 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:26.090823    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e679b58-cace-47c0-8673-3f9e9addea36" path="/var/lib/kubelet/pods/5e679b58-cace-47c0-8673-3f9e9addea36/volumes"
Nov 03 22:30:26.195814 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:26.195786    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-cbe6e297-848c8d947f-6rcr7"
Nov 03 22:30:27.202282 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:27.202253    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-cbe6e297-545ffc4c86-qnkpg"
Nov 03 22:30:27.462782 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:27.462734    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:30:27.462947 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:27.462825    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:30:27.492402 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:27.492365    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:30:28.360520 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:28.354347    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2xmvs" event={"ID":"152e2bae-faf4-4d6f-9791-01fbbe4581c3","Type":"ContainerStarted","Data":"e8f9f6e3d9852db86fb9c3d769e8f52e19eaf9377c52d1dc20b614588106c18c"}
Nov 03 22:30:28.360520 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:28.356467    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2xmvs"
Nov 03 22:30:28.364435 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:28.364400    4431 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-2xmvs container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 22:30:28.364588 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:28.364471    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2xmvs" podUID="152e2bae-faf4-4d6f-9791-01fbbe4581c3" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 22:30:28.441129 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:28.441064    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2xmvs" podStartSLOduration=1.742415885 podStartE2EDuration="4.441048823s" totalImagesPullingTime="2.698632938s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:30:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:30:28.425743188 +0000 UTC m=+1052.451599051" watchObservedRunningTime="2026-11-03 22:30:28.441048823 +0000 UTC m=+1052.466904684"
Nov 03 22:30:29.032603 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:29.032569    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:30:29.362747 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:29.362672    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2xmvs"
Nov 03 22:30:30.744211 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:30.744179    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:30:30.890702 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:30.890666    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:30:31.036701 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:31.036621    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:30:31.259903 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:30:31.259864    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:30:31.973200 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:31.973147    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:30:32.473961 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:32.473932    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:30:32.848781 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:32.848696    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:30:33.044778 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:33.033372    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:30:33.485790 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:33.485758    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:30:34.206325 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:34.206283    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:30:35.704276 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:35.704207    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:30:37.044912 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:37.044881    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:30:37.207713 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:37.207679    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:30:38.850156 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:38.850120    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:30:40.566856 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:40.566817    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:30:40.719985 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:40.719951    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:30:42.198549 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:42.198502    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5187e1/distinct-w3-5187e1-kserve-545b7fb6bc-mk49g" podUID="3ba8dc17-1b1b-4811-9dd6-595d75c6ca04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:30:42.375609 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:42.375560    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-w6-2d619f-kserve-74dbc56695-tnvtb" podUID="90310d72-c818-4fdd-b4e7-b51f633caa05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:30:42.466304 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:42.465960    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-50d738/distinct-w4-50d738-kserve-668fdf8b86-5pp9p" podUID="ce50af62-9cb6-4379-99c7-b40630538f4f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:30:42.484966 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:42.484908    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-50d738/distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r" podUID="e45b1572-6519-4c71-884a-115eeddd86cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:30:42.687242 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:42.683347    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/premium-w0-88a4d9-kserve-5559d75b7b-vqllq" podUID="05206737-4ca3-41fc-b35a-37c625cc182c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:30:42.711432 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:42.710316    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-50d738/premium-w4-50d738-kserve-59ccd7f995-xbfq2" podUID="fa0a680b-8f0d-4b19-8217-82282764511a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 03 22:30:42.734442 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:42.734332    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/simulator-w0-88a4d9-kserve-785f84888f-sjbzp" podUID="18bd460b-a0f1-415c-bcb4-41c9f70d2926" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:30:42.827487 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:42.827414    4431 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5187e1/distinct-w3-5187e1-kserve-545b7fb6bc-mk49g" podUID="3ba8dc17-1b1b-4811-9dd6-595d75c6ca04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:30:42.955333 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:42.955286    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-553753/simulator-w2-553753-kserve-64d77d4dfc-4rlbn" podUID="a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:30:43.000453 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:43.000350    4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-d6hr4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 22:30:43.000763 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:43.000713    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-d6hr4" podUID="9286b874-939f-479a-b771-a4e877cbd50f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:30:43.171858 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:43.171812    4431 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-50d738/premium-w4-50d738-kserve-59ccd7f995-xbfq2" podUID="fa0a680b-8f0d-4b19-8217-82282764511a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:30:43.207602 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:43.206327    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5187e1/distinct-w3-5187e1-kserve-545b7fb6bc-mk49g" podUID="3ba8dc17-1b1b-4811-9dd6-595d75c6ca04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:30:43.207602 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:43.207084    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5187e1/distinct-w3-5187e1-kserve-545b7fb6bc-mk49g"
Nov 03 22:30:43.208792 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:43.208763    4431 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-553753/distinct-2-w2-553753-kserve-5bf8568844-7q4l2" podUID="a1894b80-164c-4ce1-94f3-1e5011d9759e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:30:43.578238 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:43.558439    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-w6-2d619f-kserve-74dbc56695-tnvtb" podUID="90310d72-c818-4fdd-b4e7-b51f633caa05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:30:43.578238 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:43.558854    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-50d738/distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r" podUID="e45b1572-6519-4c71-884a-115eeddd86cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Nov 03 22:30:43.578238 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:43.559898    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2d619f/distinct-w6-2d619f-kserve-74dbc56695-tnvtb"
Nov 03 22:30:43.578238 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:43.559955    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-50d738/distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r"
Nov 03 22:30:43.610245 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:43.588304    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:30:43.610582 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:43.593810    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-50d738/distinct-w4-50d738-kserve-668fdf8b86-5pp9p" podUID="ce50af62-9cb6-4379-99c7-b40630538f4f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 03 22:30:43.616247 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:43.606862    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-553753/distinct-2-w2-553753-kserve-5bf8568844-7q4l2" podUID="a1894b80-164c-4ce1-94f3-1e5011d9759e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Nov 03 22:30:43.626241 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:43.617115    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-50d738/distinct-w4-50d738-kserve-668fdf8b86-5pp9p"
Nov 03 22:30:43.714071 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:43.714015    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-50d738/premium-w4-50d738-kserve-59ccd7f995-xbfq2" podUID="fa0a680b-8f0d-4b19-8217-82282764511a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:30:43.714740 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:43.714718    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-50d738/premium-w4-50d738-kserve-59ccd7f995-xbfq2"
Nov 03 22:30:43.735498 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:43.735435    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/premium-w0-88a4d9-kserve-5559d75b7b-vqllq" podUID="05206737-4ca3-41fc-b35a-37c625cc182c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:30:43.737398 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:43.737353    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-88a4d9/premium-w0-88a4d9-kserve-5559d75b7b-vqllq"
Nov 03 22:30:43.943644 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:43.739333    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/simulator-w0-88a4d9-kserve-785f84888f-sjbzp" podUID="18bd460b-a0f1-415c-bcb4-41c9f70d2926" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded"
Nov 03 22:30:43.943644 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:43.798830    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:30:43.943644 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:43.936112    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-88a4d9/simulator-w0-88a4d9-kserve-785f84888f-sjbzp"
Nov 03 22:30:44.044240 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:44.009144    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-553753/simulator-w2-553753-kserve-64d77d4dfc-4rlbn" podUID="a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:30:44.045051 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:44.045017    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-553753/simulator-w2-553753-kserve-64d77d4dfc-4rlbn"
Nov 03 22:30:44.257306 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:44.257010    4431 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-88a4d9/premium-w0-88a4d9-kserve-5559d75b7b-vqllq" podUID="05206737-4ca3-41fc-b35a-37c625cc182c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": context deadline exceeded"
Nov 03 22:30:44.362009 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:44.361956    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5187e1/distinct-w3-5187e1-kserve-545b7fb6bc-mk49g" podUID="3ba8dc17-1b1b-4811-9dd6-595d75c6ca04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Nov 03 22:30:44.390653 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:44.386316    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5187e1/premium-w3-5187e1-kserve-56bf8788d4-pp8st" podUID="a70ecfe9-020a-4331-b67c-aaaa46ed1eb1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:30:44.717389 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:44.709782    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-553753/distinct-2-w2-553753-kserve-5bf8568844-7q4l2" podUID="a1894b80-164c-4ce1-94f3-1e5011d9759e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Nov 03 22:30:44.757506 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:44.757458    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-w6-2d619f-kserve-74dbc56695-tnvtb" podUID="90310d72-c818-4fdd-b4e7-b51f633caa05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 03 22:30:44.759914 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:44.759874    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-50d738/distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r" podUID="e45b1572-6519-4c71-884a-115eeddd86cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Nov 03 22:30:44.845757 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:44.845711    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/premium-w0-88a4d9-kserve-5559d75b7b-vqllq" podUID="05206737-4ca3-41fc-b35a-37c625cc182c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 03 22:30:44.868260 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:44.863135    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-50d738/distinct-w4-50d738-kserve-668fdf8b86-5pp9p" podUID="ce50af62-9cb6-4379-99c7-b40630538f4f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:30:45.025305 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:45.002023    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/simulator-w0-88a4d9-kserve-785f84888f-sjbzp" podUID="18bd460b-a0f1-415c-bcb4-41c9f70d2926" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded"
Nov 03 22:30:45.025305 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:44.911797    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-553753/distinct-2-w2-553753-kserve-5bf8568844-7q4l2"
Nov 03 22:30:45.069484 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:45.069434    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-553753/simulator-w2-553753-kserve-64d77d4dfc-4rlbn" podUID="a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:30:45.365507 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:45.365397    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5187e1/distinct-w3-5187e1-kserve-545b7fb6bc-mk49g" podUID="3ba8dc17-1b1b-4811-9dd6-595d75c6ca04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:30:45.417238 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:45.414309    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5187e1/premium-w3-5187e1-kserve-56bf8788d4-pp8st" podUID="a70ecfe9-020a-4331-b67c-aaaa46ed1eb1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:30:45.422249 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:45.421470    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5187e1/premium-w3-5187e1-kserve-56bf8788d4-pp8st"
Nov 03 22:30:45.510296 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:45.508192    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:30:45.532159 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:45.532116    4431 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-w6-2d619f-kserve-74dbc56695-tnvtb" podUID="90310d72-c818-4fdd-b4e7-b51f633caa05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:30:45.552269 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:45.533806    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-50d738/premium-w4-50d738-kserve-59ccd7f995-xbfq2" podUID="fa0a680b-8f0d-4b19-8217-82282764511a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:30:45.785703 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:45.785665    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-88a4d9/simulator-w0-88a4d9-kserve-785f84888f-sjbzp"
Nov 03 22:30:45.796097 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:45.795047    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-553753/distinct-2-w2-553753-kserve-5bf8568844-7q4l2" podUID="a1894b80-164c-4ce1-94f3-1e5011d9759e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:30:45.821254 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:45.818572    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-50d738/distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r" podUID="e45b1572-6519-4c71-884a-115eeddd86cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:30:45.821606 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:45.821572    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/distinct-w6-2d619f-kserve-74dbc56695-tnvtb" podUID="90310d72-c818-4fdd-b4e7-b51f633caa05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 03 22:30:45.893622 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:45.893573    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/premium-w0-88a4d9-kserve-5559d75b7b-vqllq" podUID="05206737-4ca3-41fc-b35a-37c625cc182c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:30:45.979250 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:45.978396    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-50d738/distinct-w4-50d738-kserve-668fdf8b86-5pp9p" podUID="ce50af62-9cb6-4379-99c7-b40630538f4f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:30:46.027236 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:46.016883    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-50d738/distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r"
Nov 03 22:30:46.037666 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:46.037582    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-50d738/distinct-w4-50d738-kserve-668fdf8b86-5pp9p"
Nov 03 22:30:46.128668 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:46.128617    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-553753/simulator-w2-553753-kserve-64d77d4dfc-4rlbn" podUID="a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:30:46.142320 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:46.142286    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-553753/distinct-2-w2-553753-kserve-5bf8568844-7q4l2"
Nov 03 22:30:46.142593 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:46.142575    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5187e1/premium-w3-5187e1-kserve-56bf8788d4-pp8st"
Nov 03 22:30:46.142742 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:46.142728    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5187e1/distinct-w3-5187e1-kserve-545b7fb6bc-mk49g"
Nov 03 22:30:46.142881 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:46.142868    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-88a4d9/premium-w0-88a4d9-kserve-5559d75b7b-vqllq"
Nov 03 22:30:46.143014 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:46.143002    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2d619f/distinct-w6-2d619f-kserve-74dbc56695-tnvtb"
Nov 03 22:30:46.218275 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:46.182128    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:30:46.234724 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:46.234519    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-50d738/premium-w4-50d738-kserve-59ccd7f995-xbfq2"
Nov 03 22:30:46.293035 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:46.292945    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-553753/simulator-w2-553753-kserve-64d77d4dfc-4rlbn"
Nov 03 22:30:46.619918 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:46.617769    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:30:46.781978 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:46.781942    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:30:47.303245 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:47.301858    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:30:48.857189 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:48.857152    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:30:49.018251 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:49.018183    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:30:50.406724 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:50.406688    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:30:51.168209 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:51.168168    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:30:52.045394 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:52.045350    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:30:53.662182 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:53.662149    4431 generic.go:356] "Generic (PLEG): container finished" podID="e3221d3a-b814-43ec-a87a-831dd57c674d" containerID="185a55c0e354b77ac26fb94856f91ac51983c791c8230390f953571f7de2b8df" exitCode=0
Nov 03 22:30:53.662624 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:53.662232    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg" event={"ID":"e3221d3a-b814-43ec-a87a-831dd57c674d","Type":"ContainerDied","Data":"185a55c0e354b77ac26fb94856f91ac51983c791c8230390f953571f7de2b8df"}
Nov 03 22:30:53.662666 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:53.662653    4431 scope.go:122] "RemoveContainer" containerID="185a55c0e354b77ac26fb94856f91ac51983c791c8230390f953571f7de2b8df"
Nov 03 22:30:53.664368 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:53.664342    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:30:53.664495 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:53.664406    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:30:53.664495 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:53.664439    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:30:53.783712 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:53.780885    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d05919ef-98cb9cd48-4lr79"]
Nov 03 22:30:53.806496 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:53.806463    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d05919ef-5fbd9c9b67-rb26m"]
Nov 03 22:30:53.808514 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:53.808484    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:30:53.949920 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:53.949830    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:30:54.087687 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:54.087647    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:30:54.683465 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:54.683433    4431 generic.go:356] "Generic (PLEG): container finished" podID="cc93ee10-7052-448e-999d-e2f072f4ada1" containerID="1df403008efcf9dab2d96cc57805a188dbe9795cf901647eca02d31d3f1fe498" exitCode=0
Nov 03 22:30:54.683960 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:54.683466    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq" event={"ID":"cc93ee10-7052-448e-999d-e2f072f4ada1","Type":"ContainerDied","Data":"1df403008efcf9dab2d96cc57805a188dbe9795cf901647eca02d31d3f1fe498"}
Nov 03 22:30:54.684019 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:54.683979    4431 scope.go:122] "RemoveContainer" containerID="1df403008efcf9dab2d96cc57805a188dbe9795cf901647eca02d31d3f1fe498"
Nov 03 22:30:54.685825 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:54.685796    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:30:54.685969 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:54.685862    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:30:54.685969 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:54.685894    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:30:54.686088 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:54.685964    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg" event={"ID":"e3221d3a-b814-43ec-a87a-831dd57c674d","Type":"ContainerStarted","Data":"343924dcf877bbaeadfd5943f6c6fa5b4a67695f80ee9ca89aa62b494175a6a0"}
Nov 03 22:30:55.265238 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:55.264503    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg"
Nov 03 22:30:55.265238 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:55.264621    4431 patch_prober.go:29] interesting pod/e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 03 22:30:55.265238 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:55.264677    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg" podUID="e3221d3a-b814-43ec-a87a-831dd57c674d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 03 22:30:55.414608 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:55.414561    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:30:55.695979 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:55.695942    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq" event={"ID":"cc93ee10-7052-448e-999d-e2f072f4ada1","Type":"ContainerStarted","Data":"c8018975d1bd8c374957081ee571b57d9bb9feef7ccdf6c40bd811c38bce8661"}
Nov 03 22:30:56.264205 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:56.263733    4431 patch_prober.go:29] interesting pod/e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 03 22:30:56.264205 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:56.263795    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg" podUID="e3221d3a-b814-43ec-a87a-831dd57c674d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 03 22:30:56.497836 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:56.497798    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq"
Nov 03 22:30:56.498012 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:56.497925    4431 patch_prober.go:29] interesting pod/e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 03 22:30:56.498012 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:56.497969    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq" podUID="cc93ee10-7052-448e-999d-e2f072f4ada1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 03 22:30:56.933315 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:56.932439    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:30:57.262295 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:57.261673    4431 patch_prober.go:29] interesting pod/e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 03 22:30:57.262295 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:57.261734    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg" podUID="e3221d3a-b814-43ec-a87a-831dd57c674d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 03 22:30:57.405918 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:57.405880    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:30:57.497769 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:57.497732    4431 patch_prober.go:29] interesting pod/e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 03 22:30:57.497945 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:57.497782    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq" podUID="cc93ee10-7052-448e-999d-e2f072f4ada1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 03 22:30:58.279469 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:58.279357    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:30:58.280236 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:58.280108    4431 patch_prober.go:29] interesting pod/e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 03 22:30:58.280236 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:58.280165    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg" podUID="e3221d3a-b814-43ec-a87a-831dd57c674d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 03 22:30:58.505698 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:58.505660    4431 patch_prober.go:29] interesting pod/e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 03 22:30:58.505980 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:58.505951    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq" podUID="cc93ee10-7052-448e-999d-e2f072f4ada1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 03 22:30:58.742350 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:58.734123    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:30:58.783009 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:58.781700    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-d05919ef-98cb9cd48-4lr79" podUID="dc03599b-31be-47bc-a0e4-eeb8ec81e3cd" containerName="payload-processing" containerID="cri-o://06e2a91457f9255ff6179c3dd48abab329ec67c99e499418a098a9af2672dffa" gracePeriod=25
Nov 03 22:30:58.814255 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:58.807613    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-d05919ef-5fbd9c9b67-rb26m" podUID="14a18e02-91fb-4254-b687-644447f65f01" containerName="payload-pre-processing" containerID="cri-o://ef635b397f62150881a2544b616520829c3e02c7f90707b58c45518ccf330aee" gracePeriod=25
Nov 03 22:30:58.880833 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:58.880789    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:30:59.121621 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:30:59.121587    4431 logging.go:55] [core] [Channel #83 SubChannel #86]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.76:50052", ServerName: "10.133.0.76:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.76:50052: connect: connection refused"
Nov 03 22:30:59.121829 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:30:59.121593    4431 logging.go:55] [core] [Channel #84 SubChannel #85]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.77:50052", ServerName: "10.133.0.77:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.77:50052: connect: connection refused"
Nov 03 22:30:59.316889 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:59.312867    4431 patch_prober.go:29] interesting pod/e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 03 22:30:59.316889 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:59.312923    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg" podUID="e3221d3a-b814-43ec-a87a-831dd57c674d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 03 22:30:59.510914 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:59.510871    4431 patch_prober.go:29] interesting pod/e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 03 22:30:59.511181 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:59.511152    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq" podUID="cc93ee10-7052-448e-999d-e2f072f4ada1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 03 22:30:59.522805 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:59.522780    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d05919ef-98cb9cd48-4lr79"
Nov 03 22:30:59.720825 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:59.720671    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc03599b-31be-47bc-a0e4-eeb8ec81e3cd-kube-api-access-f2lzd\" (UniqueName: \"kubernetes.io/projected/dc03599b-31be-47bc-a0e4-eeb8ec81e3cd-kube-api-access-f2lzd\") pod \"dc03599b-31be-47bc-a0e4-eeb8ec81e3cd\" (UID: \"dc03599b-31be-47bc-a0e4-eeb8ec81e3cd\") "
Nov 03 22:30:59.720993 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:59.720833    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc03599b-31be-47bc-a0e4-eeb8ec81e3cd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc03599b-31be-47bc-a0e4-eeb8ec81e3cd-plugins-config-volume\") pod \"dc03599b-31be-47bc-a0e4-eeb8ec81e3cd\" (UID: \"dc03599b-31be-47bc-a0e4-eeb8ec81e3cd\") "
Nov 03 22:30:59.721829 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:59.721789    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc03599b-31be-47bc-a0e4-eeb8ec81e3cd-plugins-config-volume" pod "dc03599b-31be-47bc-a0e4-eeb8ec81e3cd" (UID: "dc03599b-31be-47bc-a0e4-eeb8ec81e3cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:30:59.725332 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:59.725306    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc03599b-31be-47bc-a0e4-eeb8ec81e3cd-kube-api-access-f2lzd" pod "dc03599b-31be-47bc-a0e4-eeb8ec81e3cd" (UID: "dc03599b-31be-47bc-a0e4-eeb8ec81e3cd"). InnerVolumeSpecName "kube-api-access-f2lzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:30:59.730905 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:59.730877    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d05919ef-5fbd9c9b67-rb26m"
Nov 03 22:30:59.801456 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:59.801422    4431 generic.go:356] "Generic (PLEG): container finished" podID="dc03599b-31be-47bc-a0e4-eeb8ec81e3cd" containerID="06e2a91457f9255ff6179c3dd48abab329ec67c99e499418a098a9af2672dffa" exitCode=0
Nov 03 22:30:59.801736 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:59.801582    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d05919ef-98cb9cd48-4lr79"
Nov 03 22:30:59.802191 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:59.801605    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d05919ef-98cb9cd48-4lr79" event={"ID":"dc03599b-31be-47bc-a0e4-eeb8ec81e3cd","Type":"ContainerDied","Data":"06e2a91457f9255ff6179c3dd48abab329ec67c99e499418a098a9af2672dffa"}
Nov 03 22:30:59.802348 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:59.802247    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d05919ef-98cb9cd48-4lr79" event={"ID":"dc03599b-31be-47bc-a0e4-eeb8ec81e3cd","Type":"ContainerDied","Data":"99061f0993105c76667f782e6c08e5ee5f995c98ab1804309278e2b17d5191be"}
Nov 03 22:30:59.802348 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:59.802274    4431 scope.go:122] "RemoveContainer" containerID="06e2a91457f9255ff6179c3dd48abab329ec67c99e499418a098a9af2672dffa"
Nov 03 22:30:59.804958 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:59.804533    4431 generic.go:356] "Generic (PLEG): container finished" podID="14a18e02-91fb-4254-b687-644447f65f01" containerID="ef635b397f62150881a2544b616520829c3e02c7f90707b58c45518ccf330aee" exitCode=0
Nov 03 22:30:59.804958 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:59.804588    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d05919ef-5fbd9c9b67-rb26m" event={"ID":"14a18e02-91fb-4254-b687-644447f65f01","Type":"ContainerDied","Data":"ef635b397f62150881a2544b616520829c3e02c7f90707b58c45518ccf330aee"}
Nov 03 22:30:59.804958 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:59.804629    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d05919ef-5fbd9c9b67-rb26m" event={"ID":"14a18e02-91fb-4254-b687-644447f65f01","Type":"ContainerDied","Data":"07eed472a9365700173f5e8f817a02cbd3f156211a17038c2abf086a0086316a"}
Nov 03 22:30:59.804958 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:59.804681    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d05919ef-5fbd9c9b67-rb26m"
Nov 03 22:30:59.810920 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:59.810897    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg"
Nov 03 22:30:59.816400 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:59.816382    4431 scope.go:122] "RemoveContainer" containerID="06e2a91457f9255ff6179c3dd48abab329ec67c99e499418a098a9af2672dffa"
Nov 03 22:30:59.816718 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:30:59.816694    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06e2a91457f9255ff6179c3dd48abab329ec67c99e499418a098a9af2672dffa\": container with ID starting with 06e2a91457f9255ff6179c3dd48abab329ec67c99e499418a098a9af2672dffa not found: ID does not exist" containerID="06e2a91457f9255ff6179c3dd48abab329ec67c99e499418a098a9af2672dffa"
Nov 03 22:30:59.816804 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:59.816731    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06e2a91457f9255ff6179c3dd48abab329ec67c99e499418a098a9af2672dffa"} err="failed to get container status \"06e2a91457f9255ff6179c3dd48abab329ec67c99e499418a098a9af2672dffa\": rpc error: code = NotFound desc = could not find container \"06e2a91457f9255ff6179c3dd48abab329ec67c99e499418a098a9af2672dffa\": container with ID starting with 06e2a91457f9255ff6179c3dd48abab329ec67c99e499418a098a9af2672dffa not found: ID does not exist"
Nov 03 22:30:59.816804 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:59.816754    4431 scope.go:122] "RemoveContainer" containerID="ef635b397f62150881a2544b616520829c3e02c7f90707b58c45518ccf330aee"
Nov 03 22:30:59.822833 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:59.822795    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2lzd\" (UniqueName: \"kubernetes.io/projected/dc03599b-31be-47bc-a0e4-eeb8ec81e3cd-kube-api-access-f2lzd\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:30:59.822833 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:59.822816    4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc03599b-31be-47bc-a0e4-eeb8ec81e3cd-plugins-config-volume\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:30:59.832953 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:59.832874    4431 scope.go:122] "RemoveContainer" containerID="ef635b397f62150881a2544b616520829c3e02c7f90707b58c45518ccf330aee"
Nov 03 22:30:59.833584 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:30:59.833563    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef635b397f62150881a2544b616520829c3e02c7f90707b58c45518ccf330aee\": container with ID starting with ef635b397f62150881a2544b616520829c3e02c7f90707b58c45518ccf330aee not found: ID does not exist" containerID="ef635b397f62150881a2544b616520829c3e02c7f90707b58c45518ccf330aee"
Nov 03 22:30:59.833668 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:59.833594    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef635b397f62150881a2544b616520829c3e02c7f90707b58c45518ccf330aee"} err="failed to get container status \"ef635b397f62150881a2544b616520829c3e02c7f90707b58c45518ccf330aee\": rpc error: code = NotFound desc = could not find container \"ef635b397f62150881a2544b616520829c3e02c7f90707b58c45518ccf330aee\": container with ID starting with ef635b397f62150881a2544b616520829c3e02c7f90707b58c45518ccf330aee not found: ID does not exist"
Nov 03 22:30:59.902512 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:59.902471    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d05919ef-98cb9cd48-4lr79"]
Nov 03 22:30:59.926910 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:59.910903    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d05919ef-98cb9cd48-4lr79"]
Nov 03 22:30:59.930981 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:59.930954    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14a18e02-91fb-4254-b687-644447f65f01-kube-api-access-t27c5\" (UniqueName: \"kubernetes.io/projected/14a18e02-91fb-4254-b687-644447f65f01-kube-api-access-t27c5\") pod \"14a18e02-91fb-4254-b687-644447f65f01\" (UID: \"14a18e02-91fb-4254-b687-644447f65f01\") "
Nov 03 22:30:59.931824 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:59.931800    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14a18e02-91fb-4254-b687-644447f65f01-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14a18e02-91fb-4254-b687-644447f65f01-plugins-config-volume\") pod \"14a18e02-91fb-4254-b687-644447f65f01\" (UID: \"14a18e02-91fb-4254-b687-644447f65f01\") "
Nov 03 22:30:59.932349 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:59.932322    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14a18e02-91fb-4254-b687-644447f65f01-plugins-config-volume" pod "14a18e02-91fb-4254-b687-644447f65f01" (UID: "14a18e02-91fb-4254-b687-644447f65f01"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:30:59.932911 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:59.932893    4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14a18e02-91fb-4254-b687-644447f65f01-plugins-config-volume\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:30:59.998254 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:30:59.995415    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14a18e02-91fb-4254-b687-644447f65f01-kube-api-access-t27c5" pod "14a18e02-91fb-4254-b687-644447f65f01" (UID: "14a18e02-91fb-4254-b687-644447f65f01"). InnerVolumeSpecName "kube-api-access-t27c5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:31:00.034798 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:00.034769    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t27c5\" (UniqueName: \"kubernetes.io/projected/14a18e02-91fb-4254-b687-644447f65f01-kube-api-access-t27c5\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:31:00.099452 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:00.099420    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc03599b-31be-47bc-a0e4-eeb8ec81e3cd" path="/var/lib/kubelet/pods/dc03599b-31be-47bc-a0e4-eeb8ec81e3cd/volumes"
Nov 03 22:31:00.121466 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:00.121425    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ait-d05919ef-98cb9cd48-4lr79" podUID="dc03599b-31be-47bc-a0e4-eeb8ec81e3cd" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.77:50052\" within 1s: context deadline exceeded"
Nov 03 22:31:00.121647 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:00.121573    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ait-d05919ef-5fbd9c9b67-rb26m" podUID="14a18e02-91fb-4254-b687-644447f65f01" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.76:50052\" within 1s: context deadline exceeded"
Nov 03 22:31:00.196371 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:00.196318    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d05919ef-5fbd9c9b67-rb26m"]
Nov 03 22:31:00.200392 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:00.200367    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d05919ef-5fbd9c9b67-rb26m"]
Nov 03 22:31:00.270294 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:00.268673    4431 patch_prober.go:29] interesting pod/e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 03 22:31:00.270294 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:00.268735    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg" podUID="e3221d3a-b814-43ec-a87a-831dd57c674d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 03 22:31:00.393815 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:00.393767    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq"]
Nov 03 22:31:00.394655 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:00.394621    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq" podUID="cc93ee10-7052-448e-999d-e2f072f4ada1" containerName="istio-proxy" containerID="cri-o://c8018975d1bd8c374957081ee571b57d9bb9feef7ccdf6c40bd811c38bce8661" gracePeriod=30
Nov 03 22:31:00.484271 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:00.482783    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:31:00.631889 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:00.629958    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:31:01.257443 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:01.257406    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:31:01.262286 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:31:01.262207    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:31:01.267074 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:01.267046    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg"
Nov 03 22:31:01.268175 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:01.268153    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg"
Nov 03 22:31:01.766236 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:01.766184    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:31:02.065550 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:02.065461    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq"
Nov 03 22:31:02.091686 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:02.091657    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14a18e02-91fb-4254-b687-644447f65f01" path="/var/lib/kubelet/pods/14a18e02-91fb-4254-b687-644447f65f01/volumes"
Nov 03 22:31:02.296831 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:02.296780    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:31:02.486727 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:02.486668    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:31:03.240380 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:03.240342    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-cbe6e297-545ffc4c86-qnkpg"]
Nov 03 22:31:03.270519 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:03.270483    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-cbe6e297-848c8d947f-6rcr7"]
Nov 03 22:31:03.405845 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:03.405811    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:31:03.548860 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:03.548763    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:31:03.906073 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:03.905988    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:31:05.510518 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:05.510438    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:31:06.202387 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:06.202357    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:31:06.347134 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:06.347102    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:31:06.925791 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:06.925748    4431 generic.go:356] "Generic (PLEG): container finished" podID="cc93ee10-7052-448e-999d-e2f072f4ada1" containerID="c8018975d1bd8c374957081ee571b57d9bb9feef7ccdf6c40bd811c38bce8661" exitCode=0
Nov 03 22:31:06.926352 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:06.926326    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq" event={"ID":"cc93ee10-7052-448e-999d-e2f072f4ada1","Type":"ContainerDied","Data":"c8018975d1bd8c374957081ee571b57d9bb9feef7ccdf6c40bd811c38bce8661"}
Nov 03 22:31:06.926571 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:06.926548    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq" event={"ID":"cc93ee10-7052-448e-999d-e2f072f4ada1","Type":"ContainerDied","Data":"5db0b34023226417971f36e2fad84a951dbddd51c02d11f5e84b8abb30657de9"}
Nov 03 22:31:06.926681 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:06.926584    4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5db0b34023226417971f36e2fad84a951dbddd51c02d11f5e84b8abb30657de9"
Nov 03 22:31:06.926681 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:06.926604    4431 scope.go:122] "RemoveContainer" containerID="1df403008efcf9dab2d96cc57805a188dbe9795cf901647eca02d31d3f1fe498"
Nov 03 22:31:06.929429 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:06.929404    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq"
Nov 03 22:31:06.948821 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:06.948785    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:31:06.955313 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:06.955281    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc93ee10-7052-448e-999d-e2f072f4ada1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc93ee10-7052-448e-999d-e2f072f4ada1-credential-socket\") pod \"cc93ee10-7052-448e-999d-e2f072f4ada1\" (UID: \"cc93ee10-7052-448e-999d-e2f072f4ada1\") "
Nov 03 22:31:06.955444 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:06.955342    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc93ee10-7052-448e-999d-e2f072f4ada1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc93ee10-7052-448e-999d-e2f072f4ada1-istio-data\") pod \"cc93ee10-7052-448e-999d-e2f072f4ada1\" (UID: \"cc93ee10-7052-448e-999d-e2f072f4ada1\") "
Nov 03 22:31:06.955444 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:06.955411    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc93ee10-7052-448e-999d-e2f072f4ada1-istio-token\" (UniqueName: \"kubernetes.io/projected/cc93ee10-7052-448e-999d-e2f072f4ada1-istio-token\") pod \"cc93ee10-7052-448e-999d-e2f072f4ada1\" (UID: \"cc93ee10-7052-448e-999d-e2f072f4ada1\") "
Nov 03 22:31:06.955569 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:06.955457    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc93ee10-7052-448e-999d-e2f072f4ada1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc93ee10-7052-448e-999d-e2f072f4ada1-istiod-ca-cert\") pod \"cc93ee10-7052-448e-999d-e2f072f4ada1\" (UID: \"cc93ee10-7052-448e-999d-e2f072f4ada1\") "
Nov 03 22:31:06.955569 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:06.955512    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc93ee10-7052-448e-999d-e2f072f4ada1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc93ee10-7052-448e-999d-e2f072f4ada1-workload-certs\") pod \"cc93ee10-7052-448e-999d-e2f072f4ada1\" (UID: \"cc93ee10-7052-448e-999d-e2f072f4ada1\") "
Nov 03 22:31:06.955569 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:06.955554    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc93ee10-7052-448e-999d-e2f072f4ada1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc93ee10-7052-448e-999d-e2f072f4ada1-istio-envoy\") pod \"cc93ee10-7052-448e-999d-e2f072f4ada1\" (UID: \"cc93ee10-7052-448e-999d-e2f072f4ada1\") "
Nov 03 22:31:06.955729 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:06.955677    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc93ee10-7052-448e-999d-e2f072f4ada1-kube-api-access-xhhll\" (UniqueName: \"kubernetes.io/projected/cc93ee10-7052-448e-999d-e2f072f4ada1-kube-api-access-xhhll\") pod \"cc93ee10-7052-448e-999d-e2f072f4ada1\" (UID: \"cc93ee10-7052-448e-999d-e2f072f4ada1\") "
Nov 03 22:31:06.955809 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:06.955793    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc93ee10-7052-448e-999d-e2f072f4ada1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc93ee10-7052-448e-999d-e2f072f4ada1-workload-socket\") pod \"cc93ee10-7052-448e-999d-e2f072f4ada1\" (UID: \"cc93ee10-7052-448e-999d-e2f072f4ada1\") "
Nov 03 22:31:06.955880 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:06.955857    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cc93ee10-7052-448e-999d-e2f072f4ada1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc93ee10-7052-448e-999d-e2f072f4ada1-istio-podinfo\") pod \"cc93ee10-7052-448e-999d-e2f072f4ada1\" (UID: \"cc93ee10-7052-448e-999d-e2f072f4ada1\") "
Nov 03 22:31:06.956812 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:06.956787    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc93ee10-7052-448e-999d-e2f072f4ada1-istiod-ca-cert" pod "cc93ee10-7052-448e-999d-e2f072f4ada1" (UID: "cc93ee10-7052-448e-999d-e2f072f4ada1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:31:06.957298 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:06.957047    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc93ee10-7052-448e-999d-e2f072f4ada1-credential-socket" pod "cc93ee10-7052-448e-999d-e2f072f4ada1" (UID: "cc93ee10-7052-448e-999d-e2f072f4ada1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:31:06.957298 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:06.957153    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc93ee10-7052-448e-999d-e2f072f4ada1-istio-data" pod "cc93ee10-7052-448e-999d-e2f072f4ada1" (UID: "cc93ee10-7052-448e-999d-e2f072f4ada1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:31:06.957623 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:06.957584    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc93ee10-7052-448e-999d-e2f072f4ada1-workload-socket" pod "cc93ee10-7052-448e-999d-e2f072f4ada1" (UID: "cc93ee10-7052-448e-999d-e2f072f4ada1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:31:06.957728 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:06.957672    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc93ee10-7052-448e-999d-e2f072f4ada1-workload-certs" pod "cc93ee10-7052-448e-999d-e2f072f4ada1" (UID: "cc93ee10-7052-448e-999d-e2f072f4ada1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:31:06.961099 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:06.959872    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc93ee10-7052-448e-999d-e2f072f4ada1-kube-api-access-xhhll" pod "cc93ee10-7052-448e-999d-e2f072f4ada1" (UID: "cc93ee10-7052-448e-999d-e2f072f4ada1"). InnerVolumeSpecName "kube-api-access-xhhll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:31:06.961703 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:06.961676    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc93ee10-7052-448e-999d-e2f072f4ada1-istio-envoy" pod "cc93ee10-7052-448e-999d-e2f072f4ada1" (UID: "cc93ee10-7052-448e-999d-e2f072f4ada1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:31:06.962194 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:06.962172    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc93ee10-7052-448e-999d-e2f072f4ada1-istio-podinfo" pod "cc93ee10-7052-448e-999d-e2f072f4ada1" (UID: "cc93ee10-7052-448e-999d-e2f072f4ada1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 22:31:06.963616 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:06.963593    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc93ee10-7052-448e-999d-e2f072f4ada1-istio-token" pod "cc93ee10-7052-448e-999d-e2f072f4ada1" (UID: "cc93ee10-7052-448e-999d-e2f072f4ada1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:31:07.058471 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:07.058433    4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc93ee10-7052-448e-999d-e2f072f4ada1-credential-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:31:07.058471 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:07.058469    4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc93ee10-7052-448e-999d-e2f072f4ada1-istio-data\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:31:07.058684 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:07.058483    4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc93ee10-7052-448e-999d-e2f072f4ada1-istio-token\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:31:07.058684 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:07.058495    4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc93ee10-7052-448e-999d-e2f072f4ada1-istiod-ca-cert\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:31:07.058684 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:07.058506    4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc93ee10-7052-448e-999d-e2f072f4ada1-workload-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:31:07.058684 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:07.058516    4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc93ee10-7052-448e-999d-e2f072f4ada1-istio-envoy\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:31:07.058684 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:07.058527    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhhll\" (UniqueName: \"kubernetes.io/projected/cc93ee10-7052-448e-999d-e2f072f4ada1-kube-api-access-xhhll\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:31:07.058684 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:07.058538    4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc93ee10-7052-448e-999d-e2f072f4ada1-workload-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:31:07.058684 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:07.058550    4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc93ee10-7052-448e-999d-e2f072f4ada1-istio-podinfo\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:31:07.391722 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:07.391633    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:31:07.540100 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:07.540009    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:31:07.707111 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:07.705630    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:31:07.935542 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:07.935514    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq"
Nov 03 22:31:08.029963 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:08.029869    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq"]
Nov 03 22:31:08.038046 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:08.038003    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-d05919ef-openshift-default-7597d49499-9f7fq"]
Nov 03 22:31:08.090108 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:08.090072    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc93ee10-7052-448e-999d-e2f072f4ada1" path="/var/lib/kubelet/pods/cc93ee10-7052-448e-999d-e2f072f4ada1/volumes"
Nov 03 22:31:08.241296 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:08.241206    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-cbe6e297-545ffc4c86-qnkpg" podUID="10ce4d4e-6f02-4163-9723-df0c2a18f7bf" containerName="payload-processing" containerID="cri-o://5541f8bf0c23047ce49346f83f9666822300206acb13772ef7ab8a8e9b91ca71" gracePeriod=25
Nov 03 22:31:08.277645 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:08.277488    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-cbe6e297-848c8d947f-6rcr7" podUID="48ef2535-a763-4dd1-9a0d-787464c05720" containerName="payload-pre-processing" containerID="cri-o://e7c41627aefd379929cd1271f7f7464d68d9ed8bf8d519a1018a639270924b53" gracePeriod=25
Nov 03 22:31:08.563752 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:08.563725    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-cbe6e297-545ffc4c86-qnkpg"
Nov 03 22:31:08.603367 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:08.603343    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-cbe6e297-848c8d947f-6rcr7"
Nov 03 22:31:08.606831 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:08.606807    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10ce4d4e-6f02-4163-9723-df0c2a18f7bf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10ce4d4e-6f02-4163-9723-df0c2a18f7bf-plugins-config-volume\") pod \"10ce4d4e-6f02-4163-9723-df0c2a18f7bf\" (UID: \"10ce4d4e-6f02-4163-9723-df0c2a18f7bf\") "
Nov 03 22:31:08.606947 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:08.606901    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10ce4d4e-6f02-4163-9723-df0c2a18f7bf-kube-api-access-gks79\" (UniqueName: \"kubernetes.io/projected/10ce4d4e-6f02-4163-9723-df0c2a18f7bf-kube-api-access-gks79\") pod \"10ce4d4e-6f02-4163-9723-df0c2a18f7bf\" (UID: \"10ce4d4e-6f02-4163-9723-df0c2a18f7bf\") "
Nov 03 22:31:08.607153 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:08.607133    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10ce4d4e-6f02-4163-9723-df0c2a18f7bf-plugins-config-volume" pod "10ce4d4e-6f02-4163-9723-df0c2a18f7bf" (UID: "10ce4d4e-6f02-4163-9723-df0c2a18f7bf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:31:08.607416 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:08.607399    4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10ce4d4e-6f02-4163-9723-df0c2a18f7bf-plugins-config-volume\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:31:08.608856 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:08.608839    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10ce4d4e-6f02-4163-9723-df0c2a18f7bf-kube-api-access-gks79" pod "10ce4d4e-6f02-4163-9723-df0c2a18f7bf" (UID: "10ce4d4e-6f02-4163-9723-df0c2a18f7bf"). InnerVolumeSpecName "kube-api-access-gks79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:31:08.673626 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:08.673589    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:31:08.708396 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:08.708366    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48ef2535-a763-4dd1-9a0d-787464c05720-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48ef2535-a763-4dd1-9a0d-787464c05720-plugins-config-volume\") pod \"48ef2535-a763-4dd1-9a0d-787464c05720\" (UID: \"48ef2535-a763-4dd1-9a0d-787464c05720\") "
Nov 03 22:31:08.708564 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:08.708550    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48ef2535-a763-4dd1-9a0d-787464c05720-kube-api-access-5p8z2\" (UniqueName: \"kubernetes.io/projected/48ef2535-a763-4dd1-9a0d-787464c05720-kube-api-access-5p8z2\") pod \"48ef2535-a763-4dd1-9a0d-787464c05720\" (UID: \"48ef2535-a763-4dd1-9a0d-787464c05720\") "
Nov 03 22:31:08.708862 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:08.708836    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48ef2535-a763-4dd1-9a0d-787464c05720-plugins-config-volume" pod "48ef2535-a763-4dd1-9a0d-787464c05720" (UID: "48ef2535-a763-4dd1-9a0d-787464c05720"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:31:08.709307 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:08.709280    4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48ef2535-a763-4dd1-9a0d-787464c05720-plugins-config-volume\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:31:08.709307 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:08.709317    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gks79\" (UniqueName: \"kubernetes.io/projected/10ce4d4e-6f02-4163-9723-df0c2a18f7bf-kube-api-access-gks79\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:31:08.710808 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:08.710785    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48ef2535-a763-4dd1-9a0d-787464c05720-kube-api-access-5p8z2" pod "48ef2535-a763-4dd1-9a0d-787464c05720" (UID: "48ef2535-a763-4dd1-9a0d-787464c05720"). InnerVolumeSpecName "kube-api-access-5p8z2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:31:08.813715 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:08.813284    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5p8z2\" (UniqueName: \"kubernetes.io/projected/48ef2535-a763-4dd1-9a0d-787464c05720-kube-api-access-5p8z2\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:31:08.907761 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:08.907674    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:31:08.952681 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:08.948309    4431 generic.go:356] "Generic (PLEG): container finished" podID="48ef2535-a763-4dd1-9a0d-787464c05720" containerID="e7c41627aefd379929cd1271f7f7464d68d9ed8bf8d519a1018a639270924b53" exitCode=0
Nov 03 22:31:08.952681 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:08.948613    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-cbe6e297-848c8d947f-6rcr7" event={"ID":"48ef2535-a763-4dd1-9a0d-787464c05720","Type":"ContainerDied","Data":"e7c41627aefd379929cd1271f7f7464d68d9ed8bf8d519a1018a639270924b53"}
Nov 03 22:31:08.952681 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:08.948639    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-cbe6e297-848c8d947f-6rcr7"
Nov 03 22:31:08.952681 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:08.948809    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-cbe6e297-848c8d947f-6rcr7" event={"ID":"48ef2535-a763-4dd1-9a0d-787464c05720","Type":"ContainerDied","Data":"fd38a7bdd7aff42c8371d1c6dfc81050705c1d31d8a1bad34eb250ba46110f51"}
Nov 03 22:31:08.952681 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:08.948862    4431 scope.go:122] "RemoveContainer" containerID="e7c41627aefd379929cd1271f7f7464d68d9ed8bf8d519a1018a639270924b53"
Nov 03 22:31:08.952681 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:08.951013    4431 generic.go:356] "Generic (PLEG): container finished" podID="10ce4d4e-6f02-4163-9723-df0c2a18f7bf" containerID="5541f8bf0c23047ce49346f83f9666822300206acb13772ef7ab8a8e9b91ca71" exitCode=0
Nov 03 22:31:08.952681 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:08.951174    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-cbe6e297-545ffc4c86-qnkpg" event={"ID":"10ce4d4e-6f02-4163-9723-df0c2a18f7bf","Type":"ContainerDied","Data":"5541f8bf0c23047ce49346f83f9666822300206acb13772ef7ab8a8e9b91ca71"}
Nov 03 22:31:08.952681 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:08.951329    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-cbe6e297-545ffc4c86-qnkpg"
Nov 03 22:31:08.952681 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:08.951337    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-cbe6e297-545ffc4c86-qnkpg" event={"ID":"10ce4d4e-6f02-4163-9723-df0c2a18f7bf","Type":"ContainerDied","Data":"e6fd7bda691ba46e4ce048e862084fe789971dfbd892aa34300c30612f08228d"}
Nov 03 22:31:08.985189 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:08.983891    4431 scope.go:122] "RemoveContainer" containerID="e7c41627aefd379929cd1271f7f7464d68d9ed8bf8d519a1018a639270924b53"
Nov 03 22:31:08.985189 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:31:08.984594    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7c41627aefd379929cd1271f7f7464d68d9ed8bf8d519a1018a639270924b53\": container with ID starting with e7c41627aefd379929cd1271f7f7464d68d9ed8bf8d519a1018a639270924b53 not found: ID does not exist" containerID="e7c41627aefd379929cd1271f7f7464d68d9ed8bf8d519a1018a639270924b53"
Nov 03 22:31:08.985189 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:08.984630    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7c41627aefd379929cd1271f7f7464d68d9ed8bf8d519a1018a639270924b53"} err="failed to get container status \"e7c41627aefd379929cd1271f7f7464d68d9ed8bf8d519a1018a639270924b53\": rpc error: code = NotFound desc = could not find container \"e7c41627aefd379929cd1271f7f7464d68d9ed8bf8d519a1018a639270924b53\": container with ID starting with e7c41627aefd379929cd1271f7f7464d68d9ed8bf8d519a1018a639270924b53 not found: ID does not exist"
Nov 03 22:31:08.985189 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:08.984652    4431 scope.go:122] "RemoveContainer" containerID="5541f8bf0c23047ce49346f83f9666822300206acb13772ef7ab8a8e9b91ca71"
Nov 03 22:31:08.996924 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:08.996899    4431 scope.go:122] "RemoveContainer" containerID="5541f8bf0c23047ce49346f83f9666822300206acb13772ef7ab8a8e9b91ca71"
Nov 03 22:31:08.998853 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:31:08.997313    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5541f8bf0c23047ce49346f83f9666822300206acb13772ef7ab8a8e9b91ca71\": container with ID starting with 5541f8bf0c23047ce49346f83f9666822300206acb13772ef7ab8a8e9b91ca71 not found: ID does not exist" containerID="5541f8bf0c23047ce49346f83f9666822300206acb13772ef7ab8a8e9b91ca71"
Nov 03 22:31:08.998853 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:08.997355    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5541f8bf0c23047ce49346f83f9666822300206acb13772ef7ab8a8e9b91ca71"} err="failed to get container status \"5541f8bf0c23047ce49346f83f9666822300206acb13772ef7ab8a8e9b91ca71\": rpc error: code = NotFound desc = could not find container \"5541f8bf0c23047ce49346f83f9666822300206acb13772ef7ab8a8e9b91ca71\": container with ID starting with 5541f8bf0c23047ce49346f83f9666822300206acb13772ef7ab8a8e9b91ca71 not found: ID does not exist"
Nov 03 22:31:09.034841 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:09.034801    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-cbe6e297-545ffc4c86-qnkpg"]
Nov 03 22:31:09.044076 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:09.044045    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-cbe6e297-545ffc4c86-qnkpg"]
Nov 03 22:31:09.083260 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:09.081632    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:31:09.088698 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:09.088655    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-cbe6e297-848c8d947f-6rcr7"]
Nov 03 22:31:09.094472 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:09.094436    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-cbe6e297-848c8d947f-6rcr7"]
Nov 03 22:31:09.235716 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:09.235682    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:31:09.461954 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:09.461921    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:31:10.092866 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:10.092502    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10ce4d4e-6f02-4163-9723-df0c2a18f7bf" path="/var/lib/kubelet/pods/10ce4d4e-6f02-4163-9723-df0c2a18f7bf/volumes"
Nov 03 22:31:10.093381 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:10.093152    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48ef2535-a763-4dd1-9a0d-787464c05720" path="/var/lib/kubelet/pods/48ef2535-a763-4dd1-9a0d-787464c05720/volumes"
Nov 03 22:31:11.035516 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:11.035456    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:31:12.663092 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:12.663054    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:31:13.618264 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:13.618222    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:31:13.770119 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:13.770062    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:31:14.310415 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:14.310374    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:31:15.240037 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:15.239997    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:31:15.401520 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:15.401486    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:31:16.088356 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:16.088319    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:31:17.711454 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:17.711419    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:31:18.436972 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:18.436934    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:31:19.442464 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:19.439325    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:31:20.986475 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:20.986436    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:31:21.135074 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:21.135023    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:31:21.611891 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:21.611854    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:31:22.937871 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:22.937838    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:31:23.209429 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:23.209335    4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-r4qjb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 22:31:23.209715 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:23.209667    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-r4qjb" podUID="d5687c54-7c44-4806-846e-3646f096b7ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:31:24.537992 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:24.537957    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:31:26.095795 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:26.095761    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:31:27.729509 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:27.729478    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:31:29.362747 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:29.362710    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:31:29.518154 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:29.518112    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:31:29.657471 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:29.657381    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:31:31.095199 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:31.095152    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:31:31.269040 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:31:31.268945    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:31:32.675439 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:32.675404    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:31:34.207355 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:34.207317    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:31:35.775783 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:35.775731    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:31:37.298731 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:37.298696    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:31:38.851428 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:38.851391    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:31:40.451504 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:40.451421    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:31:41.751954 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:41.751919    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:31:42.156774 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:42.156693    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:31:42.297046 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:42.297017    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:31:42.627428 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:42.627392    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:31:43.040659 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:43.040627    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:31:43.771850 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:43.771818    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:31:45.397458 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:45.397421    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:31:46.273992 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:46.273951    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:31:46.922193 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:46.922153    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:31:47.069875 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:47.069839    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:31:48.664401 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:48.664366    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:31:48.928288 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:48.928169    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:31:49.070090 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:49.070044    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:31:50.514350 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:50.514317    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:31:51.287766 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:51.287734    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:31:52.004280 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:52.004189    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:31:53.596463 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:53.596426    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:31:54.327072 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:54.327035    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:31:54.739610 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:54.739578    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:31:55.205765 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:55.205728    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:31:55.421744 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:55.421655    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:31:55.570418 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:55.570318    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:31:56.996557 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:56.996517    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:31:58.862401 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:58.862362    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:31:59.558985 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:59.558954    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:31:59.710169 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:31:59.710134    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:32:00.707740 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:00.707703    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:32:01.260319 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:32:01.260278    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:32:01.613611 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:01.613529    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:32:01.763063 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:01.763026    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:32:02.638858 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:02.638802    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:32:04.257822 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:04.257790    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:32:06.014019 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:06.013985    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:32:06.162977 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:06.162941    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:32:06.302063 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:06.301979    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:32:07.638524 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:07.638489    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:32:09.262999 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.262967    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84"]
Nov 03 22:32:09.263651 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.263635    4431 cpu_manager.go:414] "containerMap: removing container" podUID="14a18e02-91fb-4254-b687-644447f65f01" containerName="payload-pre-processing"
Nov 03 22:32:09.263697 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.263652    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14a18e02-91fb-4254-b687-644447f65f01" containerName="payload-pre-processing"
Nov 03 22:32:09.263697 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.263681    4431 cpu_manager.go:414] "containerMap: removing container" podUID="cc93ee10-7052-448e-999d-e2f072f4ada1" containerName="istio-proxy"
Nov 03 22:32:09.263697 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.263686    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc93ee10-7052-448e-999d-e2f072f4ada1" containerName="istio-proxy"
Nov 03 22:32:09.263697 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.263696    4431 cpu_manager.go:414] "containerMap: removing container" podUID="10ce4d4e-6f02-4163-9723-df0c2a18f7bf" containerName="payload-processing"
Nov 03 22:32:09.263825 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.263701    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10ce4d4e-6f02-4163-9723-df0c2a18f7bf" containerName="payload-processing"
Nov 03 22:32:09.263825 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.263707    4431 cpu_manager.go:414] "containerMap: removing container" podUID="cc93ee10-7052-448e-999d-e2f072f4ada1" containerName="istio-proxy"
Nov 03 22:32:09.263825 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.263712    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc93ee10-7052-448e-999d-e2f072f4ada1" containerName="istio-proxy"
Nov 03 22:32:09.263825 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.263719    4431 cpu_manager.go:414] "containerMap: removing container" podUID="dc03599b-31be-47bc-a0e4-eeb8ec81e3cd" containerName="payload-processing"
Nov 03 22:32:09.263825 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.263724    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc03599b-31be-47bc-a0e4-eeb8ec81e3cd" containerName="payload-processing"
Nov 03 22:32:09.263825 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.263729    4431 cpu_manager.go:414] "containerMap: removing container" podUID="48ef2535-a763-4dd1-9a0d-787464c05720" containerName="payload-pre-processing"
Nov 03 22:32:09.263825 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.263734    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48ef2535-a763-4dd1-9a0d-787464c05720" containerName="payload-pre-processing"
Nov 03 22:32:09.263825 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.263753    4431 cpu_manager.go:414] "containerMap: removing container" podUID="5e679b58-cace-47c0-8673-3f9e9addea36" containerName="payload-processing"
Nov 03 22:32:09.263825 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.263757    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e679b58-cace-47c0-8673-3f9e9addea36" containerName="payload-processing"
Nov 03 22:32:09.264096 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.263835    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="10ce4d4e-6f02-4163-9723-df0c2a18f7bf" containerName="payload-processing"
Nov 03 22:32:09.264096 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.263842    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc93ee10-7052-448e-999d-e2f072f4ada1" containerName="istio-proxy"
Nov 03 22:32:09.264096 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.263849    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc03599b-31be-47bc-a0e4-eeb8ec81e3cd" containerName="payload-processing"
Nov 03 22:32:09.264096 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.263856    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="48ef2535-a763-4dd1-9a0d-787464c05720" containerName="payload-pre-processing"
Nov 03 22:32:09.264096 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.263863    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e679b58-cace-47c0-8673-3f9e9addea36" containerName="payload-processing"
Nov 03 22:32:09.264096 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.263872    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="14a18e02-91fb-4254-b687-644447f65f01" containerName="payload-pre-processing"
Nov 03 22:32:09.268815 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.268794    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84"
Nov 03 22:32:09.275985 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.275954    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp"]
Nov 03 22:32:09.276976 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.276952    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc93ee10-7052-448e-999d-e2f072f4ada1" containerName="istio-proxy"
Nov 03 22:32:09.282870 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.282850    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp"
Nov 03 22:32:09.300842 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.300298    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84"]
Nov 03 22:32:09.306256 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.306202    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp"]
Nov 03 22:32:09.308834 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.308776    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:32:09.371482 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.371451    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-0847aa-openshift-default-dockercfg-dkbjf\""
Nov 03 22:32:09.387025 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.386980    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c33b591b-ff1c-4385-99df-343b21eb04a8-istio-data\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp\" (UID: \"c33b591b-ff1c-4385-99df-343b21eb04a8\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp"
Nov 03 22:32:09.387652 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.387037    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8af62636-7952-421c-8302-0af2cfefe3b7-workload-certs\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84\" (UID: \"8af62636-7952-421c-8302-0af2cfefe3b7\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84"
Nov 03 22:32:09.387652 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.387081    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8af62636-7952-421c-8302-0af2cfefe3b7-istio-token\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84\" (UID: \"8af62636-7952-421c-8302-0af2cfefe3b7\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84"
Nov 03 22:32:09.387652 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.387120    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c33b591b-ff1c-4385-99df-343b21eb04a8-workload-certs\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp\" (UID: \"c33b591b-ff1c-4385-99df-343b21eb04a8\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp"
Nov 03 22:32:09.387652 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.387253    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c33b591b-ff1c-4385-99df-343b21eb04a8-workload-socket\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp\" (UID: \"c33b591b-ff1c-4385-99df-343b21eb04a8\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp"
Nov 03 22:32:09.387652 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.387278    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8kwk\" (UniqueName: \"kubernetes.io/projected/c33b591b-ff1c-4385-99df-343b21eb04a8-kube-api-access-s8kwk\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp\" (UID: \"c33b591b-ff1c-4385-99df-343b21eb04a8\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp"
Nov 03 22:32:09.387652 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.387314    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8af62636-7952-421c-8302-0af2cfefe3b7-istio-envoy\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84\" (UID: \"8af62636-7952-421c-8302-0af2cfefe3b7\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84"
Nov 03 22:32:09.387652 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.387401    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8af62636-7952-421c-8302-0af2cfefe3b7-credential-socket\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84\" (UID: \"8af62636-7952-421c-8302-0af2cfefe3b7\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84"
Nov 03 22:32:09.387652 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.387437    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c33b591b-ff1c-4385-99df-343b21eb04a8-credential-socket\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp\" (UID: \"c33b591b-ff1c-4385-99df-343b21eb04a8\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp"
Nov 03 22:32:09.387652 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.387512    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c33b591b-ff1c-4385-99df-343b21eb04a8-istiod-ca-cert\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp\" (UID: \"c33b591b-ff1c-4385-99df-343b21eb04a8\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp"
Nov 03 22:32:09.387652 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.387537    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8af62636-7952-421c-8302-0af2cfefe3b7-istio-podinfo\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84\" (UID: \"8af62636-7952-421c-8302-0af2cfefe3b7\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84"
Nov 03 22:32:09.387652 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.387569    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c33b591b-ff1c-4385-99df-343b21eb04a8-istio-podinfo\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp\" (UID: \"c33b591b-ff1c-4385-99df-343b21eb04a8\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp"
Nov 03 22:32:09.387652 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.387621    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c33b591b-ff1c-4385-99df-343b21eb04a8-istio-envoy\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp\" (UID: \"c33b591b-ff1c-4385-99df-343b21eb04a8\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp"
Nov 03 22:32:09.387652 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.387644    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8af62636-7952-421c-8302-0af2cfefe3b7-istiod-ca-cert\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84\" (UID: \"8af62636-7952-421c-8302-0af2cfefe3b7\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84"
Nov 03 22:32:09.388151 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.387706    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r9sm\" (UniqueName: \"kubernetes.io/projected/8af62636-7952-421c-8302-0af2cfefe3b7-kube-api-access-7r9sm\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84\" (UID: \"8af62636-7952-421c-8302-0af2cfefe3b7\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84"
Nov 03 22:32:09.388151 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.387823    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c33b591b-ff1c-4385-99df-343b21eb04a8-istio-token\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp\" (UID: \"c33b591b-ff1c-4385-99df-343b21eb04a8\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp"
Nov 03 22:32:09.388151 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.387906    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8af62636-7952-421c-8302-0af2cfefe3b7-workload-socket\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84\" (UID: \"8af62636-7952-421c-8302-0af2cfefe3b7\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84"
Nov 03 22:32:09.388151 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.387959    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8af62636-7952-421c-8302-0af2cfefe3b7-istio-data\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84\" (UID: \"8af62636-7952-421c-8302-0af2cfefe3b7\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84"
Nov 03 22:32:09.489331 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.489291    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c33b591b-ff1c-4385-99df-343b21eb04a8-istio-token\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp\" (UID: \"c33b591b-ff1c-4385-99df-343b21eb04a8\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp"
Nov 03 22:32:09.489519 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.489386    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8af62636-7952-421c-8302-0af2cfefe3b7-workload-socket\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84\" (UID: \"8af62636-7952-421c-8302-0af2cfefe3b7\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84"
Nov 03 22:32:09.489519 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.489426    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8af62636-7952-421c-8302-0af2cfefe3b7-istio-data\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84\" (UID: \"8af62636-7952-421c-8302-0af2cfefe3b7\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84"
Nov 03 22:32:09.489519 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.489465    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c33b591b-ff1c-4385-99df-343b21eb04a8-istio-data\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp\" (UID: \"c33b591b-ff1c-4385-99df-343b21eb04a8\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp"
Nov 03 22:32:09.489519 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.489486    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8af62636-7952-421c-8302-0af2cfefe3b7-workload-certs\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84\" (UID: \"8af62636-7952-421c-8302-0af2cfefe3b7\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84"
Nov 03 22:32:09.489519 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.489508    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8af62636-7952-421c-8302-0af2cfefe3b7-istio-token\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84\" (UID: \"8af62636-7952-421c-8302-0af2cfefe3b7\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84"
Nov 03 22:32:09.489811 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.489535    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c33b591b-ff1c-4385-99df-343b21eb04a8-workload-certs\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp\" (UID: \"c33b591b-ff1c-4385-99df-343b21eb04a8\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp"
Nov 03 22:32:09.489811 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.489613    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c33b591b-ff1c-4385-99df-343b21eb04a8-workload-socket\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp\" (UID: \"c33b591b-ff1c-4385-99df-343b21eb04a8\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp"
Nov 03 22:32:09.489811 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.489634    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8kwk\" (UniqueName: \"kubernetes.io/projected/c33b591b-ff1c-4385-99df-343b21eb04a8-kube-api-access-s8kwk\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp\" (UID: \"c33b591b-ff1c-4385-99df-343b21eb04a8\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp"
Nov 03 22:32:09.489811 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.489657    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8af62636-7952-421c-8302-0af2cfefe3b7-istio-envoy\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84\" (UID: \"8af62636-7952-421c-8302-0af2cfefe3b7\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84"
Nov 03 22:32:09.489811 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.489717    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8af62636-7952-421c-8302-0af2cfefe3b7-credential-socket\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84\" (UID: \"8af62636-7952-421c-8302-0af2cfefe3b7\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84"
Nov 03 22:32:09.489811 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.489740    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c33b591b-ff1c-4385-99df-343b21eb04a8-credential-socket\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp\" (UID: \"c33b591b-ff1c-4385-99df-343b21eb04a8\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp"
Nov 03 22:32:09.489811 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.489785    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c33b591b-ff1c-4385-99df-343b21eb04a8-istiod-ca-cert\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp\" (UID: \"c33b591b-ff1c-4385-99df-343b21eb04a8\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp"
Nov 03 22:32:09.489811 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.489800    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8af62636-7952-421c-8302-0af2cfefe3b7-istio-podinfo\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84\" (UID: \"8af62636-7952-421c-8302-0af2cfefe3b7\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84"
Nov 03 22:32:09.490231 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.489821    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c33b591b-ff1c-4385-99df-343b21eb04a8-istio-podinfo\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp\" (UID: \"c33b591b-ff1c-4385-99df-343b21eb04a8\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp"
Nov 03 22:32:09.490231 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.489854    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c33b591b-ff1c-4385-99df-343b21eb04a8-istio-envoy\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp\" (UID: \"c33b591b-ff1c-4385-99df-343b21eb04a8\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp"
Nov 03 22:32:09.490231 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.489872    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8af62636-7952-421c-8302-0af2cfefe3b7-istiod-ca-cert\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84\" (UID: \"8af62636-7952-421c-8302-0af2cfefe3b7\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84"
Nov 03 22:32:09.490231 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.489909    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r9sm\" (UniqueName: \"kubernetes.io/projected/8af62636-7952-421c-8302-0af2cfefe3b7-kube-api-access-7r9sm\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84\" (UID: \"8af62636-7952-421c-8302-0af2cfefe3b7\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84"
Nov 03 22:32:09.490231 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.490053    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8af62636-7952-421c-8302-0af2cfefe3b7-workload-certs\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84\" (UID: \"8af62636-7952-421c-8302-0af2cfefe3b7\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84"
Nov 03 22:32:09.491070 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.491047    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c33b591b-ff1c-4385-99df-343b21eb04a8-istio-data\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp\" (UID: \"c33b591b-ff1c-4385-99df-343b21eb04a8\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp"
Nov 03 22:32:09.491765 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.491369    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c33b591b-ff1c-4385-99df-343b21eb04a8-workload-socket\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp\" (UID: \"c33b591b-ff1c-4385-99df-343b21eb04a8\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp"
Nov 03 22:32:09.492431 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.491916    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c33b591b-ff1c-4385-99df-343b21eb04a8-workload-certs\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp\" (UID: \"c33b591b-ff1c-4385-99df-343b21eb04a8\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp"
Nov 03 22:32:09.492431 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.492417    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c33b591b-ff1c-4385-99df-343b21eb04a8-credential-socket\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp\" (UID: \"c33b591b-ff1c-4385-99df-343b21eb04a8\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp"
Nov 03 22:32:09.493640 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.493607    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c33b591b-ff1c-4385-99df-343b21eb04a8-istio-podinfo\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp\" (UID: \"c33b591b-ff1c-4385-99df-343b21eb04a8\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp"
Nov 03 22:32:09.494289 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.494260    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c33b591b-ff1c-4385-99df-343b21eb04a8-istiod-ca-cert\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp\" (UID: \"c33b591b-ff1c-4385-99df-343b21eb04a8\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp"
Nov 03 22:32:09.494397 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.494373    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8af62636-7952-421c-8302-0af2cfefe3b7-workload-socket\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84\" (UID: \"8af62636-7952-421c-8302-0af2cfefe3b7\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84"
Nov 03 22:32:09.494513 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.494489    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c33b591b-ff1c-4385-99df-343b21eb04a8-istio-envoy\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp\" (UID: \"c33b591b-ff1c-4385-99df-343b21eb04a8\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp"
Nov 03 22:32:09.494642 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.494622    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8af62636-7952-421c-8302-0af2cfefe3b7-credential-socket\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84\" (UID: \"8af62636-7952-421c-8302-0af2cfefe3b7\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84"
Nov 03 22:32:09.494882 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.494851    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8af62636-7952-421c-8302-0af2cfefe3b7-istio-data\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84\" (UID: \"8af62636-7952-421c-8302-0af2cfefe3b7\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84"
Nov 03 22:32:09.495101 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.495074    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8af62636-7952-421c-8302-0af2cfefe3b7-istiod-ca-cert\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84\" (UID: \"8af62636-7952-421c-8302-0af2cfefe3b7\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84"
Nov 03 22:32:09.500958 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.500921    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8af62636-7952-421c-8302-0af2cfefe3b7-istio-podinfo\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84\" (UID: \"8af62636-7952-421c-8302-0af2cfefe3b7\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84"
Nov 03 22:32:09.503465 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.503437    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8af62636-7952-421c-8302-0af2cfefe3b7-istio-envoy\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84\" (UID: \"8af62636-7952-421c-8302-0af2cfefe3b7\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84"
Nov 03 22:32:09.512669 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.512534    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c33b591b-ff1c-4385-99df-343b21eb04a8-istio-token\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp\" (UID: \"c33b591b-ff1c-4385-99df-343b21eb04a8\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp"
Nov 03 22:32:09.513464 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.513398    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r9sm\" (UniqueName: \"kubernetes.io/projected/8af62636-7952-421c-8302-0af2cfefe3b7-kube-api-access-7r9sm\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84\" (UID: \"8af62636-7952-421c-8302-0af2cfefe3b7\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84"
Nov 03 22:32:09.514319 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.514296    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8af62636-7952-421c-8302-0af2cfefe3b7-istio-token\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84\" (UID: \"8af62636-7952-421c-8302-0af2cfefe3b7\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84"
Nov 03 22:32:09.521495 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.521474    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8kwk\" (UniqueName: \"kubernetes.io/projected/c33b591b-ff1c-4385-99df-343b21eb04a8-kube-api-access-s8kwk\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp\" (UID: \"c33b591b-ff1c-4385-99df-343b21eb04a8\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp"
Nov 03 22:32:09.590282 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.590246    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84"
Nov 03 22:32:09.612869 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.612842    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp"
Nov 03 22:32:09.918363 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.918332    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84"]
Nov 03 22:32:09.918605 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:32:09.918583    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8af62636_7952_421c_8302_0af2cfefe3b7.slice/crio-31a8df8d60cc16e761fc7da94548d2b9d9d1908dc91aff6ceae490963e7691ed WatchSource:0}: Error finding container 31a8df8d60cc16e761fc7da94548d2b9d9d1908dc91aff6ceae490963e7691ed: Status 404 returned error can't find the container with id 31a8df8d60cc16e761fc7da94548d2b9d9d1908dc91aff6ceae490963e7691ed
Nov 03 22:32:09.921524 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.921493    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:32:09.921613 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.921559    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:32:09.921613 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.921585    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:32:09.945027 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.945000    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp"]
Nov 03 22:32:09.945461 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:32:09.945435    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc33b591b_ff1c_4385_99df_343b21eb04a8.slice/crio-476b78671e9bfb93d264fbeb5ed85384c9f55717501f058085b679a5d0e6a9e4 WatchSource:0}: Error finding container 476b78671e9bfb93d264fbeb5ed85384c9f55717501f058085b679a5d0e6a9e4: Status 404 returned error can't find the container with id 476b78671e9bfb93d264fbeb5ed85384c9f55717501f058085b679a5d0e6a9e4
Nov 03 22:32:09.948628 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.948595    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:32:09.948703 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.948659    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:32:09.948703 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:09.948690    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:32:10.523979 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:10.523943    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84" event={"ID":"8af62636-7952-421c-8302-0af2cfefe3b7","Type":"ContainerStarted","Data":"220a438bc6516a2c859e660e15fc26b7846372be746b740b240b71297a6ae1ff"}
Nov 03 22:32:10.524460 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:10.524000    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84" event={"ID":"8af62636-7952-421c-8302-0af2cfefe3b7","Type":"ContainerStarted","Data":"31a8df8d60cc16e761fc7da94548d2b9d9d1908dc91aff6ceae490963e7691ed"}
Nov 03 22:32:10.525551 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:10.525524    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp" event={"ID":"c33b591b-ff1c-4385-99df-343b21eb04a8","Type":"ContainerStarted","Data":"a3361c43861b84348017c36a619439ed2938bc8bc85a25cda75748552a860184"}
Nov 03 22:32:10.525671 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:10.525572    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp" event={"ID":"c33b591b-ff1c-4385-99df-343b21eb04a8","Type":"ContainerStarted","Data":"476b78671e9bfb93d264fbeb5ed85384c9f55717501f058085b679a5d0e6a9e4"}
Nov 03 22:32:10.581250 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:10.581151    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84" podStartSLOduration=1.5811282009999998 podStartE2EDuration="1.581128201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:32:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:32:10.579567854 +0000 UTC m=+1154.605423717" watchObservedRunningTime="2026-11-03 22:32:10.581128201 +0000 UTC m=+1154.606984064"
Nov 03 22:32:10.596594 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:10.596553    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84"
Nov 03 22:32:10.598033 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:10.597998    4431 patch_prober.go:29] interesting pod/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 22:32:10.598183 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:10.598057    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84" podUID="8af62636-7952-421c-8302-0af2cfefe3b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 22:32:10.613472 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:10.613444    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp"
Nov 03 22:32:10.630645 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:10.630590    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp" podStartSLOduration=1.630574733 podStartE2EDuration="1.630574733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:32:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:32:10.626243944 +0000 UTC m=+1154.652099799" watchObservedRunningTime="2026-11-03 22:32:10.630574733 +0000 UTC m=+1154.656430595"
Nov 03 22:32:10.854295 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:10.853973    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:32:11.592240 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:11.591785    4431 patch_prober.go:29] interesting pod/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 22:32:11.592240 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:11.591848    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84" podUID="8af62636-7952-421c-8302-0af2cfefe3b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 22:32:11.615274 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:11.615246    4431 patch_prober.go:29] interesting pod/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 03 22:32:11.615485 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:11.615442    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp" podUID="c33b591b-ff1c-4385-99df-343b21eb04a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 03 22:32:12.367240 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:12.366948    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:32:12.559654 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:12.558743    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:32:12.598239 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:12.590998    4431 patch_prober.go:29] interesting pod/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 22:32:12.598239 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:12.591061    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84" podUID="8af62636-7952-421c-8302-0af2cfefe3b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 22:32:12.616062 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:12.615832    4431 patch_prober.go:29] interesting pod/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 03 22:32:12.616062 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:12.615893    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp" podUID="c33b591b-ff1c-4385-99df-343b21eb04a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 03 22:32:13.592107 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:13.591736    4431 patch_prober.go:29] interesting pod/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 22:32:13.592107 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:13.591833    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84" podUID="8af62636-7952-421c-8302-0af2cfefe3b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 22:32:13.613585 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:13.613556    4431 patch_prober.go:29] interesting pod/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 03 22:32:13.613946 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:13.613598    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp" podUID="c33b591b-ff1c-4385-99df-343b21eb04a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 03 22:32:14.213357 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:14.213317    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:32:14.592757 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:14.592663    4431 patch_prober.go:29] interesting pod/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 22:32:14.592757 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:14.592717    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84" podUID="8af62636-7952-421c-8302-0af2cfefe3b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 22:32:14.614094 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:14.614056    4431 patch_prober.go:29] interesting pod/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 03 22:32:14.614524 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:14.614116    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp" podUID="c33b591b-ff1c-4385-99df-343b21eb04a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 03 22:32:14.787234 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:14.786591    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:32:14.925201 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:14.925126    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:32:15.591732 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:15.591692    4431 patch_prober.go:29] interesting pod/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 22:32:15.592019 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:15.591987    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84" podUID="8af62636-7952-421c-8302-0af2cfefe3b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 22:32:15.616907 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:15.616872    4431 patch_prober.go:29] interesting pod/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 03 22:32:15.617381 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:15.617355    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp" podUID="c33b591b-ff1c-4385-99df-343b21eb04a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 03 22:32:15.799267 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:15.760735    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:32:16.400881 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:16.400837    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:32:16.613973 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:16.613937    4431 patch_prober.go:29] interesting pod/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 03 22:32:16.614325 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:16.614005    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp" podUID="c33b591b-ff1c-4385-99df-343b21eb04a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 03 22:32:16.708558 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:16.708518    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84"
Nov 03 22:32:16.710247 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:16.710192    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84"
Nov 03 22:32:16.717000 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:16.716957    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84"
Nov 03 22:32:17.618396 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:17.618361    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp"
Nov 03 22:32:17.618779 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:17.618758    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp"
Nov 03 22:32:17.619546 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:17.619525    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp"
Nov 03 22:32:17.896471 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:17.896392    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:32:18.276709 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:18.276677    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:32:18.418440 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:18.418403    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:32:18.559508 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:18.559425    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:32:18.914923 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:18.914885    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:32:19.935360 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:19.935309    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:32:21.627598 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:21.625748    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:32:23.456507 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:23.456475    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:32:25.067122 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:25.067080    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:32:25.262133 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:25.262082    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:32:26.686320 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:26.686284    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:32:28.282131 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:28.282097    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:32:29.834998 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:29.834960    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:32:31.260180 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:32:31.260143    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:32:31.431800 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:31.431755    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:32:32.322588 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:32.322533    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:32:33.020940 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:33.020907    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:32:34.591873 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:34.591838    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:32:36.492859 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:36.492820    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:32:36.642690 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:36.642651    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:32:36.783075 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:36.782926    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:32:37.087633 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:37.087521    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:32:38.738724 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:38.738649    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:32:40.370000 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:40.369969    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:32:42.016252 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:42.016199    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:32:43.499055 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:43.498997    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:32:43.665711 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:43.665657    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:32:43.795668 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:43.795548    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg"]
Nov 03 22:32:43.796112 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:43.796078    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg" podUID="e3221d3a-b814-43ec-a87a-831dd57c674d" containerName="istio-proxy" containerID="cri-o://343924dcf877bbaeadfd5943f6c6fa5b4a67695f80ee9ca89aa62b494175a6a0" gracePeriod=30
Nov 03 22:32:43.857381 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:43.857344    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:32:44.649566 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:44.649534    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:32:45.195424 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:45.195386    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:32:45.336323 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:45.336284    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:32:46.173795 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:46.173764    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:32:46.280342 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:46.280280    4431 patch_prober.go:29] interesting pod/e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 22:32:46.280533 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:46.280350    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg" podUID="e3221d3a-b814-43ec-a87a-831dd57c674d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 22:32:46.334884 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:46.334845    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:32:46.482719 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:46.482621    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:32:46.873507 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:46.873419    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:32:47.021039 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:47.020993    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:32:47.496637 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:47.496606    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:32:48.800939 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:48.800830    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:32:48.965267 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:48.965177    4431 generic.go:356] "Generic (PLEG): container finished" podID="e3221d3a-b814-43ec-a87a-831dd57c674d" containerID="343924dcf877bbaeadfd5943f6c6fa5b4a67695f80ee9ca89aa62b494175a6a0" exitCode=0
Nov 03 22:32:48.965267 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:48.965263    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg" event={"ID":"e3221d3a-b814-43ec-a87a-831dd57c674d","Type":"ContainerDied","Data":"343924dcf877bbaeadfd5943f6c6fa5b4a67695f80ee9ca89aa62b494175a6a0"}
Nov 03 22:32:48.965523 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:48.965430    4431 scope.go:122] "RemoveContainer" containerID="185a55c0e354b77ac26fb94856f91ac51983c791c8230390f953571f7de2b8df"
Nov 03 22:32:48.991481 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:48.991416    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:32:49.150630 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:49.150590    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg"
Nov 03 22:32:49.161059 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:49.161033    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3221d3a-b814-43ec-a87a-831dd57c674d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3221d3a-b814-43ec-a87a-831dd57c674d-workload-socket\") pod \"e3221d3a-b814-43ec-a87a-831dd57c674d\" (UID: \"e3221d3a-b814-43ec-a87a-831dd57c674d\") "
Nov 03 22:32:49.161175 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:49.161098    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3221d3a-b814-43ec-a87a-831dd57c674d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3221d3a-b814-43ec-a87a-831dd57c674d-istiod-ca-cert\") pod \"e3221d3a-b814-43ec-a87a-831dd57c674d\" (UID: \"e3221d3a-b814-43ec-a87a-831dd57c674d\") "
Nov 03 22:32:49.161175 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:49.161136    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e3221d3a-b814-43ec-a87a-831dd57c674d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3221d3a-b814-43ec-a87a-831dd57c674d-istio-podinfo\") pod \"e3221d3a-b814-43ec-a87a-831dd57c674d\" (UID: \"e3221d3a-b814-43ec-a87a-831dd57c674d\") "
Nov 03 22:32:49.161175 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:49.161163    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3221d3a-b814-43ec-a87a-831dd57c674d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3221d3a-b814-43ec-a87a-831dd57c674d-credential-socket\") pod \"e3221d3a-b814-43ec-a87a-831dd57c674d\" (UID: \"e3221d3a-b814-43ec-a87a-831dd57c674d\") "
Nov 03 22:32:49.161321 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:49.161191    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3221d3a-b814-43ec-a87a-831dd57c674d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3221d3a-b814-43ec-a87a-831dd57c674d-istio-data\") pod \"e3221d3a-b814-43ec-a87a-831dd57c674d\" (UID: \"e3221d3a-b814-43ec-a87a-831dd57c674d\") "
Nov 03 22:32:49.161321 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:49.161249    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3221d3a-b814-43ec-a87a-831dd57c674d-kube-api-access-k2kp8\" (UniqueName: \"kubernetes.io/projected/e3221d3a-b814-43ec-a87a-831dd57c674d-kube-api-access-k2kp8\") pod \"e3221d3a-b814-43ec-a87a-831dd57c674d\" (UID: \"e3221d3a-b814-43ec-a87a-831dd57c674d\") "
Nov 03 22:32:49.161321 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:49.161285    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3221d3a-b814-43ec-a87a-831dd57c674d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3221d3a-b814-43ec-a87a-831dd57c674d-workload-certs\") pod \"e3221d3a-b814-43ec-a87a-831dd57c674d\" (UID: \"e3221d3a-b814-43ec-a87a-831dd57c674d\") "
Nov 03 22:32:49.161492 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:49.161353    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3221d3a-b814-43ec-a87a-831dd57c674d-workload-socket" pod "e3221d3a-b814-43ec-a87a-831dd57c674d" (UID: "e3221d3a-b814-43ec-a87a-831dd57c674d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:32:49.161492 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:49.161452    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3221d3a-b814-43ec-a87a-831dd57c674d-istio-token\" (UniqueName: \"kubernetes.io/projected/e3221d3a-b814-43ec-a87a-831dd57c674d-istio-token\") pod \"e3221d3a-b814-43ec-a87a-831dd57c674d\" (UID: \"e3221d3a-b814-43ec-a87a-831dd57c674d\") "
Nov 03 22:32:49.161605 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:49.161517    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3221d3a-b814-43ec-a87a-831dd57c674d-istio-data" pod "e3221d3a-b814-43ec-a87a-831dd57c674d" (UID: "e3221d3a-b814-43ec-a87a-831dd57c674d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:32:49.161605 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:49.161526    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3221d3a-b814-43ec-a87a-831dd57c674d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3221d3a-b814-43ec-a87a-831dd57c674d-istio-envoy\") pod \"e3221d3a-b814-43ec-a87a-831dd57c674d\" (UID: \"e3221d3a-b814-43ec-a87a-831dd57c674d\") "
Nov 03 22:32:49.161723 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:49.161620    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3221d3a-b814-43ec-a87a-831dd57c674d-istiod-ca-cert" pod "e3221d3a-b814-43ec-a87a-831dd57c674d" (UID: "e3221d3a-b814-43ec-a87a-831dd57c674d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:32:49.161723 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:49.161702    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3221d3a-b814-43ec-a87a-831dd57c674d-workload-certs" pod "e3221d3a-b814-43ec-a87a-831dd57c674d" (UID: "e3221d3a-b814-43ec-a87a-831dd57c674d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:32:49.161829 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:49.161736    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3221d3a-b814-43ec-a87a-831dd57c674d-credential-socket" pod "e3221d3a-b814-43ec-a87a-831dd57c674d" (UID: "e3221d3a-b814-43ec-a87a-831dd57c674d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:32:49.162303 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:49.162280    4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3221d3a-b814-43ec-a87a-831dd57c674d-credential-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:32:49.162303 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:49.162301    4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3221d3a-b814-43ec-a87a-831dd57c674d-istio-data\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:32:49.162472 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:49.162311    4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3221d3a-b814-43ec-a87a-831dd57c674d-workload-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:32:49.162472 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:49.162318    4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3221d3a-b814-43ec-a87a-831dd57c674d-workload-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:32:49.162472 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:49.162329    4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3221d3a-b814-43ec-a87a-831dd57c674d-istiod-ca-cert\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:32:49.164119 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:49.164097    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3221d3a-b814-43ec-a87a-831dd57c674d-istio-token" pod "e3221d3a-b814-43ec-a87a-831dd57c674d" (UID: "e3221d3a-b814-43ec-a87a-831dd57c674d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:32:49.164831 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:49.164812    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3221d3a-b814-43ec-a87a-831dd57c674d-istio-envoy" pod "e3221d3a-b814-43ec-a87a-831dd57c674d" (UID: "e3221d3a-b814-43ec-a87a-831dd57c674d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:32:49.164956 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:49.164937    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3221d3a-b814-43ec-a87a-831dd57c674d-kube-api-access-k2kp8" pod "e3221d3a-b814-43ec-a87a-831dd57c674d" (UID: "e3221d3a-b814-43ec-a87a-831dd57c674d"). InnerVolumeSpecName "kube-api-access-k2kp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:32:49.165297 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:49.165278    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e3221d3a-b814-43ec-a87a-831dd57c674d-istio-podinfo" pod "e3221d3a-b814-43ec-a87a-831dd57c674d" (UID: "e3221d3a-b814-43ec-a87a-831dd57c674d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 22:32:49.265234 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:49.265179    4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3221d3a-b814-43ec-a87a-831dd57c674d-istio-podinfo\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:32:49.265433 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:49.265277    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2kp8\" (UniqueName: \"kubernetes.io/projected/e3221d3a-b814-43ec-a87a-831dd57c674d-kube-api-access-k2kp8\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:32:49.265433 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:49.265294    4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3221d3a-b814-43ec-a87a-831dd57c674d-istio-token\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:32:49.265433 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:49.265307    4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3221d3a-b814-43ec-a87a-831dd57c674d-istio-envoy\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:32:49.756835 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:49.756795    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:32:49.908971 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:49.908939    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:32:49.972889 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:49.972857    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg"
Nov 03 22:32:49.973081 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:49.972858    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg" event={"ID":"e3221d3a-b814-43ec-a87a-831dd57c674d","Type":"ContainerDied","Data":"dd48b41b565cf08be92bbdb5bee91937a77e5304741700de9ac2785e85114c4f"}
Nov 03 22:32:49.973081 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:49.973041    4431 scope.go:122] "RemoveContainer" containerID="343924dcf877bbaeadfd5943f6c6fa5b4a67695f80ee9ca89aa62b494175a6a0"
Nov 03 22:32:50.027125 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:50.027046    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg"]
Nov 03 22:32:50.034125 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:50.034091    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-53879c92-openshift-default-67dfffdb77-c47vg"]
Nov 03 22:32:50.089519 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:50.089488    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3221d3a-b814-43ec-a87a-831dd57c674d" path="/var/lib/kubelet/pods/e3221d3a-b814-43ec-a87a-831dd57c674d/volumes"
Nov 03 22:32:50.515870 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:50.515836    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:32:50.767535 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:50.767501    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:32:50.918342 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:50.918302    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:32:51.551431 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:51.551397    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:32:51.694764 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:51.694732    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-23d281-97b8bbd66-7dgnl"]
Nov 03 22:32:51.695462 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:51.695441    4431 cpu_manager.go:414] "containerMap: removing container" podUID="e3221d3a-b814-43ec-a87a-831dd57c674d" containerName="istio-proxy"
Nov 03 22:32:51.695462 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:51.695462    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3221d3a-b814-43ec-a87a-831dd57c674d" containerName="istio-proxy"
Nov 03 22:32:51.695680 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:51.695666    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3221d3a-b814-43ec-a87a-831dd57c674d" containerName="istio-proxy"
Nov 03 22:32:51.700709 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:51.700685    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-23d281-97b8bbd66-7dgnl"
Nov 03 22:32:51.715558 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:51.715533    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-23d281-97b8bbd66-7dgnl"]
Nov 03 22:32:51.721315 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:51.721289    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-23d281\""
Nov 03 22:32:51.728458 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:51.728437    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:32:51.728640 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:51.728619    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-23d281-8599fb6555-qtrjc"]
Nov 03 22:32:51.729679 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:51.729657    4431 cpu_manager.go:414] "containerMap: removing container" podUID="e3221d3a-b814-43ec-a87a-831dd57c674d" containerName="istio-proxy"
Nov 03 22:32:51.729741 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:51.729683    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3221d3a-b814-43ec-a87a-831dd57c674d" containerName="istio-proxy"
Nov 03 22:32:51.729862 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:51.729849    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3221d3a-b814-43ec-a87a-831dd57c674d" containerName="istio-proxy"
Nov 03 22:32:51.733973 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:51.733955    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-23d281-8599fb6555-qtrjc"
Nov 03 22:32:51.749462 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:51.749438    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-23d281-dockercfg-br6zq\""
Nov 03 22:32:51.754530 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:51.754503    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-23d281-8599fb6555-qtrjc"]
Nov 03 22:32:51.802899 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:51.802814    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2452e573-42a1-40b4-b86f-3678286375bc-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-23d281-97b8bbd66-7dgnl\" (UID: \"2452e573-42a1-40b4-b86f-3678286375bc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-23d281-97b8bbd66-7dgnl"
Nov 03 22:32:51.803062 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:51.802973    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wt4v2\" (UniqueName: \"kubernetes.io/projected/2452e573-42a1-40b4-b86f-3678286375bc-kube-api-access-wt4v2\") pod \"payload-processing-e2e-shared-a-w3-23d281-97b8bbd66-7dgnl\" (UID: \"2452e573-42a1-40b4-b86f-3678286375bc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-23d281-97b8bbd66-7dgnl"
Nov 03 22:32:51.803062 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:51.803017    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77hll\" (UniqueName: \"kubernetes.io/projected/98293f18-47d0-49fe-8ec9-f6b80342985d-kube-api-access-77hll\") pod \"payload-pre-processing-e2e-shared-a-w3-23d281-8599fb6555-qtrjc\" (UID: \"98293f18-47d0-49fe-8ec9-f6b80342985d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-23d281-8599fb6555-qtrjc"
Nov 03 22:32:51.803188 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:51.803163    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98293f18-47d0-49fe-8ec9-f6b80342985d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-23d281-8599fb6555-qtrjc\" (UID: \"98293f18-47d0-49fe-8ec9-f6b80342985d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-23d281-8599fb6555-qtrjc"
Nov 03 22:32:51.906893 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:51.906830    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2452e573-42a1-40b4-b86f-3678286375bc-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-23d281-97b8bbd66-7dgnl\" (UID: \"2452e573-42a1-40b4-b86f-3678286375bc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-23d281-97b8bbd66-7dgnl"
Nov 03 22:32:51.907102 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:51.907014    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wt4v2\" (UniqueName: \"kubernetes.io/projected/2452e573-42a1-40b4-b86f-3678286375bc-kube-api-access-wt4v2\") pod \"payload-processing-e2e-shared-a-w3-23d281-97b8bbd66-7dgnl\" (UID: \"2452e573-42a1-40b4-b86f-3678286375bc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-23d281-97b8bbd66-7dgnl"
Nov 03 22:32:51.907102 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:51.907086    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77hll\" (UniqueName: \"kubernetes.io/projected/98293f18-47d0-49fe-8ec9-f6b80342985d-kube-api-access-77hll\") pod \"payload-pre-processing-e2e-shared-a-w3-23d281-8599fb6555-qtrjc\" (UID: \"98293f18-47d0-49fe-8ec9-f6b80342985d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-23d281-8599fb6555-qtrjc"
Nov 03 22:32:51.907262 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:51.907179    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98293f18-47d0-49fe-8ec9-f6b80342985d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-23d281-8599fb6555-qtrjc\" (UID: \"98293f18-47d0-49fe-8ec9-f6b80342985d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-23d281-8599fb6555-qtrjc"
Nov 03 22:32:51.908733 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:51.908579    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98293f18-47d0-49fe-8ec9-f6b80342985d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-23d281-8599fb6555-qtrjc\" (UID: \"98293f18-47d0-49fe-8ec9-f6b80342985d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-23d281-8599fb6555-qtrjc"
Nov 03 22:32:51.908921 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:51.908783    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2452e573-42a1-40b4-b86f-3678286375bc-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-23d281-97b8bbd66-7dgnl\" (UID: \"2452e573-42a1-40b4-b86f-3678286375bc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-23d281-97b8bbd66-7dgnl"
Nov 03 22:32:51.930611 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:51.930579    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77hll\" (UniqueName: \"kubernetes.io/projected/98293f18-47d0-49fe-8ec9-f6b80342985d-kube-api-access-77hll\") pod \"payload-pre-processing-e2e-shared-a-w3-23d281-8599fb6555-qtrjc\" (UID: \"98293f18-47d0-49fe-8ec9-f6b80342985d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-23d281-8599fb6555-qtrjc"
Nov 03 22:32:51.931989 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:51.931957    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wt4v2\" (UniqueName: \"kubernetes.io/projected/2452e573-42a1-40b4-b86f-3678286375bc-kube-api-access-wt4v2\") pod \"payload-processing-e2e-shared-a-w3-23d281-97b8bbd66-7dgnl\" (UID: \"2452e573-42a1-40b4-b86f-3678286375bc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-23d281-97b8bbd66-7dgnl"
Nov 03 22:32:52.021160 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:52.021120    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-23d281-97b8bbd66-7dgnl"
Nov 03 22:32:52.050445 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:52.050411    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-23d281-8599fb6555-qtrjc"
Nov 03 22:32:52.361512 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:52.361476    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-23d281-97b8bbd66-7dgnl"]
Nov 03 22:32:52.362064 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:32:52.362024    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2452e573_42a1_40b4_b86f_3678286375bc.slice/crio-8cec14553c18152ee52c4f4aa1f6b7406d895fe880087495a0d0d6f4caa38ccc WatchSource:0}: Error finding container 8cec14553c18152ee52c4f4aa1f6b7406d895fe880087495a0d0d6f4caa38ccc: Status 404 returned error can't find the container with id 8cec14553c18152ee52c4f4aa1f6b7406d895fe880087495a0d0d6f4caa38ccc
Nov 03 22:32:52.402821 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:32:52.402784    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98293f18_47d0_49fe_8ec9_f6b80342985d.slice/crio-b6df291bf775b70b0549e600889c510de72352b11a99eb1c38bf86ed52c344f7 WatchSource:0}: Error finding container b6df291bf775b70b0549e600889c510de72352b11a99eb1c38bf86ed52c344f7: Status 404 returned error can't find the container with id b6df291bf775b70b0549e600889c510de72352b11a99eb1c38bf86ed52c344f7
Nov 03 22:32:52.403743 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:52.403626    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-23d281-8599fb6555-qtrjc"]
Nov 03 22:32:52.428283 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:52.428254    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:32:52.585242 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:52.575748    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:32:52.727002 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:52.726961    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:32:53.000442 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:53.000350    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-23d281-97b8bbd66-7dgnl" event={"ID":"2452e573-42a1-40b4-b86f-3678286375bc","Type":"ContainerStarted","Data":"936af0258adcab8270e0bc34efc0e1efc8bbc52867a87acbb0672153ba8166b5"}
Nov 03 22:32:53.000442 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:53.000411    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-23d281-97b8bbd66-7dgnl" event={"ID":"2452e573-42a1-40b4-b86f-3678286375bc","Type":"ContainerStarted","Data":"8cec14553c18152ee52c4f4aa1f6b7406d895fe880087495a0d0d6f4caa38ccc"}
Nov 03 22:32:53.000442 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:53.000438    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-23d281-97b8bbd66-7dgnl"
Nov 03 22:32:53.002093 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:53.002070    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-23d281-8599fb6555-qtrjc" event={"ID":"98293f18-47d0-49fe-8ec9-f6b80342985d","Type":"ContainerStarted","Data":"ebb4c16b23f14fab56166d51e222dfa982a1de721a6477c3008471e91ddc1cfb"}
Nov 03 22:32:53.002209 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:53.002107    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-23d281-8599fb6555-qtrjc" event={"ID":"98293f18-47d0-49fe-8ec9-f6b80342985d","Type":"ContainerStarted","Data":"b6df291bf775b70b0549e600889c510de72352b11a99eb1c38bf86ed52c344f7"}
Nov 03 22:32:53.002209 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:53.002141    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-23d281-8599fb6555-qtrjc"
Nov 03 22:32:53.034653 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:53.034622    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:32:53.043940 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:53.043891    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-23d281-97b8bbd66-7dgnl" podStartSLOduration=2.043877196 podStartE2EDuration="2.043877196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:32:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:32:53.043294505 +0000 UTC m=+1197.069150368" watchObservedRunningTime="2026-11-03 22:32:53.043877196 +0000 UTC m=+1197.069733058"
Nov 03 22:32:53.085945 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:53.085895    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-23d281-8599fb6555-qtrjc" podStartSLOduration=2.085881267 podStartE2EDuration="2.085881267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:32:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:32:53.082599617 +0000 UTC m=+1197.108455478" watchObservedRunningTime="2026-11-03 22:32:53.085881267 +0000 UTC m=+1197.111737129"
Nov 03 22:32:53.577881 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:53.577806    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:32:53.765243 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:53.744581    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:32:53.911294 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:53.910740    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:32:54.067670 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:54.067634    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:32:54.235687 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:54.235570    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:32:54.402698 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:54.402664    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:32:54.465161 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:54.465127    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8b1efeb6-5d7dd8ff94-q446n"]
Nov 03 22:32:54.470822 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:54.470795    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8b1efeb6-68787c86c9-wpjjk"]
Nov 03 22:32:54.472384 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:54.471102    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8b1efeb6-5d7dd8ff94-q446n"
Nov 03 22:32:54.476431 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:54.476403    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8b1efeb6-68787c86c9-wpjjk"
Nov 03 22:32:54.479526 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:54.479498    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7"]
Nov 03 22:32:54.584291 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:54.537841    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-8b1efeb6-dockercfg-p6xgl\""
Nov 03 22:32:54.584291 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:54.538002    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-8b1efeb6\""
Nov 03 22:32:54.587286 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:54.587243    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/993058fc-2d6c-4c46-bf06-3aa2442b34f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8b1efeb6-68787c86c9-wpjjk\" (UID: \"993058fc-2d6c-4c46-bf06-3aa2442b34f7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8b1efeb6-68787c86c9-wpjjk"
Nov 03 22:32:54.587443 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:54.587328    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46fa8602-cba7-4489-a2ab-57d7bdecec1a-plugins-config-volume\") pod \"payload-processing-e2e-mt-8b1efeb6-5d7dd8ff94-q446n\" (UID: \"46fa8602-cba7-4489-a2ab-57d7bdecec1a\") " pod="openshift-ingress/payload-processing-e2e-mt-8b1efeb6-5d7dd8ff94-q446n"
Nov 03 22:32:54.588051 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:54.587776    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9rcp\" (UniqueName: \"kubernetes.io/projected/993058fc-2d6c-4c46-bf06-3aa2442b34f7-kube-api-access-g9rcp\") pod \"payload-pre-processing-e2e-mt-8b1efeb6-68787c86c9-wpjjk\" (UID: \"993058fc-2d6c-4c46-bf06-3aa2442b34f7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8b1efeb6-68787c86c9-wpjjk"
Nov 03 22:32:54.588051 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:54.587830    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9h9s\" (UniqueName: \"kubernetes.io/projected/46fa8602-cba7-4489-a2ab-57d7bdecec1a-kube-api-access-x9h9s\") pod \"payload-processing-e2e-mt-8b1efeb6-5d7dd8ff94-q446n\" (UID: \"46fa8602-cba7-4489-a2ab-57d7bdecec1a\") " pod="openshift-ingress/payload-processing-e2e-mt-8b1efeb6-5d7dd8ff94-q446n"
Nov 03 22:32:54.696752 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:54.696715    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9rcp\" (UniqueName: \"kubernetes.io/projected/993058fc-2d6c-4c46-bf06-3aa2442b34f7-kube-api-access-g9rcp\") pod \"payload-pre-processing-e2e-mt-8b1efeb6-68787c86c9-wpjjk\" (UID: \"993058fc-2d6c-4c46-bf06-3aa2442b34f7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8b1efeb6-68787c86c9-wpjjk"
Nov 03 22:32:54.703283 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:54.703253    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9h9s\" (UniqueName: \"kubernetes.io/projected/46fa8602-cba7-4489-a2ab-57d7bdecec1a-kube-api-access-x9h9s\") pod \"payload-processing-e2e-mt-8b1efeb6-5d7dd8ff94-q446n\" (UID: \"46fa8602-cba7-4489-a2ab-57d7bdecec1a\") " pod="openshift-ingress/payload-processing-e2e-mt-8b1efeb6-5d7dd8ff94-q446n"
Nov 03 22:32:54.703724 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:54.703694    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/993058fc-2d6c-4c46-bf06-3aa2442b34f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8b1efeb6-68787c86c9-wpjjk\" (UID: \"993058fc-2d6c-4c46-bf06-3aa2442b34f7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8b1efeb6-68787c86c9-wpjjk"
Nov 03 22:32:54.705035 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:54.705013    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46fa8602-cba7-4489-a2ab-57d7bdecec1a-plugins-config-volume\") pod \"payload-processing-e2e-mt-8b1efeb6-5d7dd8ff94-q446n\" (UID: \"46fa8602-cba7-4489-a2ab-57d7bdecec1a\") " pod="openshift-ingress/payload-processing-e2e-mt-8b1efeb6-5d7dd8ff94-q446n"
Nov 03 22:32:54.706579 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:54.706532    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46fa8602-cba7-4489-a2ab-57d7bdecec1a-plugins-config-volume\") pod \"payload-processing-e2e-mt-8b1efeb6-5d7dd8ff94-q446n\" (UID: \"46fa8602-cba7-4489-a2ab-57d7bdecec1a\") " pod="openshift-ingress/payload-processing-e2e-mt-8b1efeb6-5d7dd8ff94-q446n"
Nov 03 22:32:54.712274 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:54.712235    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/993058fc-2d6c-4c46-bf06-3aa2442b34f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8b1efeb6-68787c86c9-wpjjk\" (UID: \"993058fc-2d6c-4c46-bf06-3aa2442b34f7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8b1efeb6-68787c86c9-wpjjk"
Nov 03 22:32:54.738096 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:54.737169    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7"
Nov 03 22:32:54.776317 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:54.776285    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8b1efeb6-5d7dd8ff94-q446n"]
Nov 03 22:32:54.776561 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:54.776543    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8b1efeb6-68787c86c9-wpjjk"]
Nov 03 22:32:54.794417 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:54.788117    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7"]
Nov 03 22:32:54.802928 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:54.802884    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9h9s\" (UniqueName: \"kubernetes.io/projected/46fa8602-cba7-4489-a2ab-57d7bdecec1a-kube-api-access-x9h9s\") pod \"payload-processing-e2e-mt-8b1efeb6-5d7dd8ff94-q446n\" (UID: \"46fa8602-cba7-4489-a2ab-57d7bdecec1a\") " pod="openshift-ingress/payload-processing-e2e-mt-8b1efeb6-5d7dd8ff94-q446n"
Nov 03 22:32:54.805584 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:54.805545    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9rcp\" (UniqueName: \"kubernetes.io/projected/993058fc-2d6c-4c46-bf06-3aa2442b34f7-kube-api-access-g9rcp\") pod \"payload-pre-processing-e2e-mt-8b1efeb6-68787c86c9-wpjjk\" (UID: \"993058fc-2d6c-4c46-bf06-3aa2442b34f7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8b1efeb6-68787c86c9-wpjjk"
Nov 03 22:32:54.806988 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:54.806960    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6634b1a8-4b77-4281-8566-3470ecf3022f-workload-certs\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7\" (UID: \"6634b1a8-4b77-4281-8566-3470ecf3022f\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7"
Nov 03 22:32:54.807210 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:54.807193    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6634b1a8-4b77-4281-8566-3470ecf3022f-istiod-ca-cert\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7\" (UID: \"6634b1a8-4b77-4281-8566-3470ecf3022f\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7"
Nov 03 22:32:54.807509 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:54.807490    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6634b1a8-4b77-4281-8566-3470ecf3022f-istio-podinfo\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7\" (UID: \"6634b1a8-4b77-4281-8566-3470ecf3022f\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7"
Nov 03 22:32:54.807689 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:54.807672    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6634b1a8-4b77-4281-8566-3470ecf3022f-workload-socket\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7\" (UID: \"6634b1a8-4b77-4281-8566-3470ecf3022f\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7"
Nov 03 22:32:54.807850 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:54.807832    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6634b1a8-4b77-4281-8566-3470ecf3022f-istio-data\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7\" (UID: \"6634b1a8-4b77-4281-8566-3470ecf3022f\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7"
Nov 03 22:32:54.808132 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:54.808114    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6634b1a8-4b77-4281-8566-3470ecf3022f-istio-token\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7\" (UID: \"6634b1a8-4b77-4281-8566-3470ecf3022f\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7"
Nov 03 22:32:54.808361 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:54.808337    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6634b1a8-4b77-4281-8566-3470ecf3022f-istio-envoy\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7\" (UID: \"6634b1a8-4b77-4281-8566-3470ecf3022f\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7"
Nov 03 22:32:54.808452 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:54.808399    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6634b1a8-4b77-4281-8566-3470ecf3022f-credential-socket\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7\" (UID: \"6634b1a8-4b77-4281-8566-3470ecf3022f\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7"
Nov 03 22:32:54.808452 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:54.808431    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj95d\" (UniqueName: \"kubernetes.io/projected/6634b1a8-4b77-4281-8566-3470ecf3022f-kube-api-access-jj95d\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7\" (UID: \"6634b1a8-4b77-4281-8566-3470ecf3022f\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7"
Nov 03 22:32:54.909978 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:54.909888    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6634b1a8-4b77-4281-8566-3470ecf3022f-workload-socket\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7\" (UID: \"6634b1a8-4b77-4281-8566-3470ecf3022f\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7"
Nov 03 22:32:54.909978 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:54.909930    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6634b1a8-4b77-4281-8566-3470ecf3022f-istio-data\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7\" (UID: \"6634b1a8-4b77-4281-8566-3470ecf3022f\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7"
Nov 03 22:32:54.910249 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:54.910062    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6634b1a8-4b77-4281-8566-3470ecf3022f-istio-token\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7\" (UID: \"6634b1a8-4b77-4281-8566-3470ecf3022f\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7"
Nov 03 22:32:54.910249 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:54.910117    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6634b1a8-4b77-4281-8566-3470ecf3022f-istio-envoy\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7\" (UID: \"6634b1a8-4b77-4281-8566-3470ecf3022f\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7"
Nov 03 22:32:54.910249 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:54.910166    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6634b1a8-4b77-4281-8566-3470ecf3022f-credential-socket\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7\" (UID: \"6634b1a8-4b77-4281-8566-3470ecf3022f\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7"
Nov 03 22:32:54.910249 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:54.910193    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj95d\" (UniqueName: \"kubernetes.io/projected/6634b1a8-4b77-4281-8566-3470ecf3022f-kube-api-access-jj95d\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7\" (UID: \"6634b1a8-4b77-4281-8566-3470ecf3022f\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7"
Nov 03 22:32:54.910464 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:54.910344    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6634b1a8-4b77-4281-8566-3470ecf3022f-workload-certs\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7\" (UID: \"6634b1a8-4b77-4281-8566-3470ecf3022f\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7"
Nov 03 22:32:54.910464 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:54.910397    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6634b1a8-4b77-4281-8566-3470ecf3022f-istiod-ca-cert\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7\" (UID: \"6634b1a8-4b77-4281-8566-3470ecf3022f\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7"
Nov 03 22:32:54.910464 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:54.910400    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6634b1a8-4b77-4281-8566-3470ecf3022f-workload-socket\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7\" (UID: \"6634b1a8-4b77-4281-8566-3470ecf3022f\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7"
Nov 03 22:32:54.910615 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:54.910555    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6634b1a8-4b77-4281-8566-3470ecf3022f-istio-podinfo\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7\" (UID: \"6634b1a8-4b77-4281-8566-3470ecf3022f\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7"
Nov 03 22:32:54.911063 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:54.911018    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6634b1a8-4b77-4281-8566-3470ecf3022f-workload-certs\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7\" (UID: \"6634b1a8-4b77-4281-8566-3470ecf3022f\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7"
Nov 03 22:32:54.911516 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:54.911473    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6634b1a8-4b77-4281-8566-3470ecf3022f-istio-data\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7\" (UID: \"6634b1a8-4b77-4281-8566-3470ecf3022f\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7"
Nov 03 22:32:54.911629 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:54.911579    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6634b1a8-4b77-4281-8566-3470ecf3022f-istiod-ca-cert\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7\" (UID: \"6634b1a8-4b77-4281-8566-3470ecf3022f\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7"
Nov 03 22:32:54.911786 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:54.911755    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6634b1a8-4b77-4281-8566-3470ecf3022f-credential-socket\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7\" (UID: \"6634b1a8-4b77-4281-8566-3470ecf3022f\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7"
Nov 03 22:32:54.913088 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:54.913063    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6634b1a8-4b77-4281-8566-3470ecf3022f-istio-envoy\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7\" (UID: \"6634b1a8-4b77-4281-8566-3470ecf3022f\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7"
Nov 03 22:32:54.914008 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:54.913987    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6634b1a8-4b77-4281-8566-3470ecf3022f-istio-podinfo\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7\" (UID: \"6634b1a8-4b77-4281-8566-3470ecf3022f\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7"
Nov 03 22:32:54.919932 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:54.919907    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8b1efeb6-5d7dd8ff94-q446n"
Nov 03 22:32:54.933974 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:54.933894    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj95d\" (UniqueName: \"kubernetes.io/projected/6634b1a8-4b77-4281-8566-3470ecf3022f-kube-api-access-jj95d\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7\" (UID: \"6634b1a8-4b77-4281-8566-3470ecf3022f\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7"
Nov 03 22:32:54.934343 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:54.934048    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6634b1a8-4b77-4281-8566-3470ecf3022f-istio-token\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7\" (UID: \"6634b1a8-4b77-4281-8566-3470ecf3022f\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7"
Nov 03 22:32:55.025046 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:55.024635    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8b1efeb6-68787c86c9-wpjjk"
Nov 03 22:32:55.090370 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:55.087712    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7"
Nov 03 22:32:55.257741 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:55.257710    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8b1efeb6-5d7dd8ff94-q446n"]
Nov 03 22:32:55.406369 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:55.406340    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8b1efeb6-68787c86c9-wpjjk"]
Nov 03 22:32:55.406558 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:32:55.406536    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod993058fc_2d6c_4c46_bf06_3aa2442b34f7.slice/crio-44bcc8dd1036284f2047b965a74216005f8ba54b6e9d1bdbfd121b25832c19ad WatchSource:0}: Error finding container 44bcc8dd1036284f2047b965a74216005f8ba54b6e9d1bdbfd121b25832c19ad: Status 404 returned error can't find the container with id 44bcc8dd1036284f2047b965a74216005f8ba54b6e9d1bdbfd121b25832c19ad
Nov 03 22:32:55.414562 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:55.414478    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq"]
Nov 03 22:32:55.420421 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:55.420400    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq"
Nov 03 22:32:55.436493 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:55.436472    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-a940d5-openshift-default-dockercfg-9b7lq\""
Nov 03 22:32:55.447753 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:55.447727    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq"]
Nov 03 22:32:55.510011 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:32:55.509981    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6634b1a8_4b77_4281_8566_3470ecf3022f.slice/crio-43f6bb2e6160edf20256c4013cf4ca89f0c2990057b0417ad413370f7cb4c7b5 WatchSource:0}: Error finding container 43f6bb2e6160edf20256c4013cf4ca89f0c2990057b0417ad413370f7cb4c7b5: Status 404 returned error can't find the container with id 43f6bb2e6160edf20256c4013cf4ca89f0c2990057b0417ad413370f7cb4c7b5
Nov 03 22:32:55.516881 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:55.516844    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:32:55.516983 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:55.516933    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:32:55.516983 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:55.516976    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:32:55.517524 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:55.517492    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7"]
Nov 03 22:32:55.535381 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:55.535355    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a49eb997-50b7-45cf-a401-19c1a2b7af77-workload-certs\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq\" (UID: \"a49eb997-50b7-45cf-a401-19c1a2b7af77\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq"
Nov 03 22:32:55.535496 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:55.535406    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a49eb997-50b7-45cf-a401-19c1a2b7af77-workload-socket\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq\" (UID: \"a49eb997-50b7-45cf-a401-19c1a2b7af77\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq"
Nov 03 22:32:55.535596 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:55.535569    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2k8z\" (UniqueName: \"kubernetes.io/projected/a49eb997-50b7-45cf-a401-19c1a2b7af77-kube-api-access-z2k8z\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq\" (UID: \"a49eb997-50b7-45cf-a401-19c1a2b7af77\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq"
Nov 03 22:32:55.535655 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:55.535616    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a49eb997-50b7-45cf-a401-19c1a2b7af77-istio-podinfo\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq\" (UID: \"a49eb997-50b7-45cf-a401-19c1a2b7af77\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq"
Nov 03 22:32:55.535715 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:55.535672    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a49eb997-50b7-45cf-a401-19c1a2b7af77-credential-socket\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq\" (UID: \"a49eb997-50b7-45cf-a401-19c1a2b7af77\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq"
Nov 03 22:32:55.535773 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:55.535740    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a49eb997-50b7-45cf-a401-19c1a2b7af77-istiod-ca-cert\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq\" (UID: \"a49eb997-50b7-45cf-a401-19c1a2b7af77\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq"
Nov 03 22:32:55.535830 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:55.535796    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a49eb997-50b7-45cf-a401-19c1a2b7af77-istio-token\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq\" (UID: \"a49eb997-50b7-45cf-a401-19c1a2b7af77\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq"
Nov 03 22:32:55.535887 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:55.535827    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a49eb997-50b7-45cf-a401-19c1a2b7af77-istio-data\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq\" (UID: \"a49eb997-50b7-45cf-a401-19c1a2b7af77\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq"
Nov 03 22:32:55.535945 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:55.535893    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a49eb997-50b7-45cf-a401-19c1a2b7af77-istio-envoy\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq\" (UID: \"a49eb997-50b7-45cf-a401-19c1a2b7af77\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq"
Nov 03 22:32:55.637719 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:55.637684    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a49eb997-50b7-45cf-a401-19c1a2b7af77-workload-certs\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq\" (UID: \"a49eb997-50b7-45cf-a401-19c1a2b7af77\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq"
Nov 03 22:32:55.638033 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:55.637740    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a49eb997-50b7-45cf-a401-19c1a2b7af77-workload-socket\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq\" (UID: \"a49eb997-50b7-45cf-a401-19c1a2b7af77\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq"
Nov 03 22:32:55.638033 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:55.637829    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2k8z\" (UniqueName: \"kubernetes.io/projected/a49eb997-50b7-45cf-a401-19c1a2b7af77-kube-api-access-z2k8z\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq\" (UID: \"a49eb997-50b7-45cf-a401-19c1a2b7af77\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq"
Nov 03 22:32:55.638033 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:55.637856    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a49eb997-50b7-45cf-a401-19c1a2b7af77-istio-podinfo\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq\" (UID: \"a49eb997-50b7-45cf-a401-19c1a2b7af77\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq"
Nov 03 22:32:55.638033 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:55.637910    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a49eb997-50b7-45cf-a401-19c1a2b7af77-credential-socket\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq\" (UID: \"a49eb997-50b7-45cf-a401-19c1a2b7af77\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq"
Nov 03 22:32:55.638033 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:55.637980    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a49eb997-50b7-45cf-a401-19c1a2b7af77-istiod-ca-cert\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq\" (UID: \"a49eb997-50b7-45cf-a401-19c1a2b7af77\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq"
Nov 03 22:32:55.638033 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:55.638034    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a49eb997-50b7-45cf-a401-19c1a2b7af77-istio-token\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq\" (UID: \"a49eb997-50b7-45cf-a401-19c1a2b7af77\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq"
Nov 03 22:32:55.638450 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:55.638064    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a49eb997-50b7-45cf-a401-19c1a2b7af77-istio-data\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq\" (UID: \"a49eb997-50b7-45cf-a401-19c1a2b7af77\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq"
Nov 03 22:32:55.638450 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:55.638126    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a49eb997-50b7-45cf-a401-19c1a2b7af77-istio-envoy\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq\" (UID: \"a49eb997-50b7-45cf-a401-19c1a2b7af77\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq"
Nov 03 22:32:55.638450 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:55.638298    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a49eb997-50b7-45cf-a401-19c1a2b7af77-workload-socket\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq\" (UID: \"a49eb997-50b7-45cf-a401-19c1a2b7af77\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq"
Nov 03 22:32:55.638615 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:55.638599    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a49eb997-50b7-45cf-a401-19c1a2b7af77-workload-certs\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq\" (UID: \"a49eb997-50b7-45cf-a401-19c1a2b7af77\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq"
Nov 03 22:32:55.638948 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:55.638875    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a49eb997-50b7-45cf-a401-19c1a2b7af77-credential-socket\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq\" (UID: \"a49eb997-50b7-45cf-a401-19c1a2b7af77\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq"
Nov 03 22:32:55.640285 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:55.639863    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a49eb997-50b7-45cf-a401-19c1a2b7af77-istiod-ca-cert\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq\" (UID: \"a49eb997-50b7-45cf-a401-19c1a2b7af77\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq"
Nov 03 22:32:55.640285 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:55.640210    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a49eb997-50b7-45cf-a401-19c1a2b7af77-istio-data\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq\" (UID: \"a49eb997-50b7-45cf-a401-19c1a2b7af77\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq"
Nov 03 22:32:55.641168 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:55.641149    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a49eb997-50b7-45cf-a401-19c1a2b7af77-istio-podinfo\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq\" (UID: \"a49eb997-50b7-45cf-a401-19c1a2b7af77\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq"
Nov 03 22:32:55.641328 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:55.641303    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a49eb997-50b7-45cf-a401-19c1a2b7af77-istio-envoy\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq\" (UID: \"a49eb997-50b7-45cf-a401-19c1a2b7af77\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq"
Nov 03 22:32:55.681174 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:55.681100    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a49eb997-50b7-45cf-a401-19c1a2b7af77-istio-token\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq\" (UID: \"a49eb997-50b7-45cf-a401-19c1a2b7af77\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq"
Nov 03 22:32:55.681324 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:55.681183    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2k8z\" (UniqueName: \"kubernetes.io/projected/a49eb997-50b7-45cf-a401-19c1a2b7af77-kube-api-access-z2k8z\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq\" (UID: \"a49eb997-50b7-45cf-a401-19c1a2b7af77\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq"
Nov 03 22:32:55.735833 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:55.735801    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq"
Nov 03 22:32:56.033541 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:56.033444    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7" event={"ID":"6634b1a8-4b77-4281-8566-3470ecf3022f","Type":"ContainerStarted","Data":"8c9ec34de308f6be5a9d4ca11032917bbe1543513b0793d8895af9374c1d92ff"}
Nov 03 22:32:56.033541 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:56.033516    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7" event={"ID":"6634b1a8-4b77-4281-8566-3470ecf3022f","Type":"ContainerStarted","Data":"43f6bb2e6160edf20256c4013cf4ca89f0c2990057b0417ad413370f7cb4c7b5"}
Nov 03 22:32:56.035239 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:56.035200    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8b1efeb6-5d7dd8ff94-q446n" event={"ID":"46fa8602-cba7-4489-a2ab-57d7bdecec1a","Type":"ContainerStarted","Data":"5db57f3589454d021eb4f28470fee0ac5947220c6c2e42db0014ceb2514bff7e"}
Nov 03 22:32:56.035328 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:56.035266    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-8b1efeb6-5d7dd8ff94-q446n"
Nov 03 22:32:56.035328 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:56.035291    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8b1efeb6-5d7dd8ff94-q446n" event={"ID":"46fa8602-cba7-4489-a2ab-57d7bdecec1a","Type":"ContainerStarted","Data":"fbbe0ed204f348067b2037417961359e204d23b1d4b9649038daae422583357f"}
Nov 03 22:32:56.036919 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:56.036896    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8b1efeb6-68787c86c9-wpjjk" event={"ID":"993058fc-2d6c-4c46-bf06-3aa2442b34f7","Type":"ContainerStarted","Data":"2e61a010a432fb082e26ef45efe0aaa669cdc1f16f330161534e97c827e96ca8"}
Nov 03 22:32:56.037016 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:56.036942    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-8b1efeb6-68787c86c9-wpjjk"
Nov 03 22:32:56.037016 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:56.036958    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8b1efeb6-68787c86c9-wpjjk" event={"ID":"993058fc-2d6c-4c46-bf06-3aa2442b34f7","Type":"ContainerStarted","Data":"44bcc8dd1036284f2047b965a74216005f8ba54b6e9d1bdbfd121b25832c19ad"}
Nov 03 22:32:56.071583 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:56.071549    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq"]
Nov 03 22:32:56.073858 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:32:56.073818    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda49eb997_50b7_45cf_a401_19c1a2b7af77.slice/crio-f3a43c7bdba36889a3f321e4422fcc1a44aa11192db249bc5bc8e1b4976cd271 WatchSource:0}: Error finding container f3a43c7bdba36889a3f321e4422fcc1a44aa11192db249bc5bc8e1b4976cd271: Status 404 returned error can't find the container with id f3a43c7bdba36889a3f321e4422fcc1a44aa11192db249bc5bc8e1b4976cd271
Nov 03 22:32:56.076627 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:56.076591    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:32:56.076740 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:56.076676    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:32:56.076740 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:56.076721    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:32:56.085552 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:56.085501    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7" podStartSLOduration=2.085487097 podStartE2EDuration="2.085487097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:32:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:32:56.080420216 +0000 UTC m=+1200.106276079" watchObservedRunningTime="2026-11-03 22:32:56.085487097 +0000 UTC m=+1200.111342958"
Nov 03 22:32:56.095091 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:56.095064    4431 patch_prober.go:29] interesting pod/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Nov 03 22:32:56.095583 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:56.095554    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7" podUID="6634b1a8-4b77-4281-8566-3470ecf3022f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Nov 03 22:32:56.096557 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:56.096532    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7"
Nov 03 22:32:56.129984 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:56.129439    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-8b1efeb6-68787c86c9-wpjjk" podStartSLOduration=2.129421081 podStartE2EDuration="2.129421081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:32:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:32:56.128855047 +0000 UTC m=+1200.154710911" watchObservedRunningTime="2026-11-03 22:32:56.129421081 +0000 UTC m=+1200.155276947"
Nov 03 22:32:56.181933 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:56.181853    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-8b1efeb6-5d7dd8ff94-q446n" podStartSLOduration=2.181833715 podStartE2EDuration="2.181833715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:32:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:32:56.174461978 +0000 UTC m=+1200.200317846" watchObservedRunningTime="2026-11-03 22:32:56.181833715 +0000 UTC m=+1200.207689577"
Nov 03 22:32:56.344543 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:56.334671    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:32:56.393393 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:56.389399    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mx5q8_ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea/ovn-acl-logging/0.log"
Nov 03 22:32:56.396704 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:56.396667    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mx5q8_ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea/ovn-acl-logging/0.log"
Nov 03 22:32:57.085240 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:57.084768    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq" event={"ID":"a49eb997-50b7-45cf-a401-19c1a2b7af77","Type":"ContainerStarted","Data":"fee49d0f08b5fdd10883faed8493c368ae1af1057309fe7a270c4a7d7a7f86cf"}
Nov 03 22:32:57.085240 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:57.084866    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq" event={"ID":"a49eb997-50b7-45cf-a401-19c1a2b7af77","Type":"ContainerStarted","Data":"f3a43c7bdba36889a3f321e4422fcc1a44aa11192db249bc5bc8e1b4976cd271"}
Nov 03 22:32:57.093649 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:57.093618    4431 patch_prober.go:29] interesting pod/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Nov 03 22:32:57.093816 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:57.093667    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7" podUID="6634b1a8-4b77-4281-8566-3470ecf3022f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Nov 03 22:32:57.196037 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:57.195964    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq" podStartSLOduration=2.195942983 podStartE2EDuration="2.195942983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:32:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:32:57.180720403 +0000 UTC m=+1201.206576278" watchObservedRunningTime="2026-11-03 22:32:57.195942983 +0000 UTC m=+1201.221798846"
Nov 03 22:32:57.736752 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:57.736715    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq"
Nov 03 22:32:57.738072 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:57.738047    4431 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Nov 03 22:32:57.738201 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:57.738090    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq" podUID="a49eb997-50b7-45cf-a401-19c1a2b7af77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Nov 03 22:32:57.894091 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:57.894048    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6"]
Nov 03 22:32:57.899937 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:57.899912    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6"]
Nov 03 22:32:57.900114 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:57.900029    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6"
Nov 03 22:32:57.914632 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:57.914610    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-04988c-openshift-default-dockercfg-6kp47\""
Nov 03 22:32:58.013635 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:58.013541    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dde620e-e87f-4f87-984e-6358a642a100-credential-socket\") pod \"e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6\" (UID: \"7dde620e-e87f-4f87-984e-6358a642a100\") " pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6"
Nov 03 22:32:58.014101 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:58.014080    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdt8m\" (UniqueName: \"kubernetes.io/projected/7dde620e-e87f-4f87-984e-6358a642a100-kube-api-access-wdt8m\") pod \"e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6\" (UID: \"7dde620e-e87f-4f87-984e-6358a642a100\") " pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6"
Nov 03 22:32:58.014249 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:58.014234    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dde620e-e87f-4f87-984e-6358a642a100-workload-socket\") pod \"e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6\" (UID: \"7dde620e-e87f-4f87-984e-6358a642a100\") " pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6"
Nov 03 22:32:58.014430 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:58.014414    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dde620e-e87f-4f87-984e-6358a642a100-istio-envoy\") pod \"e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6\" (UID: \"7dde620e-e87f-4f87-984e-6358a642a100\") " pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6"
Nov 03 22:32:58.014633 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:58.014619    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dde620e-e87f-4f87-984e-6358a642a100-istio-token\") pod \"e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6\" (UID: \"7dde620e-e87f-4f87-984e-6358a642a100\") " pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6"
Nov 03 22:32:58.014756 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:58.014744    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dde620e-e87f-4f87-984e-6358a642a100-istio-data\") pod \"e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6\" (UID: \"7dde620e-e87f-4f87-984e-6358a642a100\") " pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6"
Nov 03 22:32:58.014857 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:58.014844    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dde620e-e87f-4f87-984e-6358a642a100-istiod-ca-cert\") pod \"e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6\" (UID: \"7dde620e-e87f-4f87-984e-6358a642a100\") " pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6"
Nov 03 22:32:58.015012 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:58.014998    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dde620e-e87f-4f87-984e-6358a642a100-workload-certs\") pod \"e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6\" (UID: \"7dde620e-e87f-4f87-984e-6358a642a100\") " pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6"
Nov 03 22:32:58.015134 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:58.015121    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dde620e-e87f-4f87-984e-6358a642a100-istio-podinfo\") pod \"e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6\" (UID: \"7dde620e-e87f-4f87-984e-6358a642a100\") " pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6"
Nov 03 22:32:58.092805 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:58.092768    4431 patch_prober.go:29] interesting pod/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Nov 03 22:32:58.093773 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:58.093398    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7" podUID="6634b1a8-4b77-4281-8566-3470ecf3022f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Nov 03 22:32:58.115988 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:58.115961    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdt8m\" (UniqueName: \"kubernetes.io/projected/7dde620e-e87f-4f87-984e-6358a642a100-kube-api-access-wdt8m\") pod \"e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6\" (UID: \"7dde620e-e87f-4f87-984e-6358a642a100\") " pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6"
Nov 03 22:32:58.116123 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:58.116007    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dde620e-e87f-4f87-984e-6358a642a100-workload-socket\") pod \"e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6\" (UID: \"7dde620e-e87f-4f87-984e-6358a642a100\") " pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6"
Nov 03 22:32:58.116123 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:58.116073    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dde620e-e87f-4f87-984e-6358a642a100-istio-envoy\") pod \"e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6\" (UID: \"7dde620e-e87f-4f87-984e-6358a642a100\") " pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6"
Nov 03 22:32:58.116271 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:58.116132    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dde620e-e87f-4f87-984e-6358a642a100-istio-token\") pod \"e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6\" (UID: \"7dde620e-e87f-4f87-984e-6358a642a100\") " pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6"
Nov 03 22:32:58.116271 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:58.116164    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dde620e-e87f-4f87-984e-6358a642a100-istio-data\") pod \"e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6\" (UID: \"7dde620e-e87f-4f87-984e-6358a642a100\") " pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6"
Nov 03 22:32:58.116271 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:58.116181    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dde620e-e87f-4f87-984e-6358a642a100-istiod-ca-cert\") pod \"e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6\" (UID: \"7dde620e-e87f-4f87-984e-6358a642a100\") " pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6"
Nov 03 22:32:58.116415 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:58.116292    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dde620e-e87f-4f87-984e-6358a642a100-workload-certs\") pod \"e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6\" (UID: \"7dde620e-e87f-4f87-984e-6358a642a100\") " pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6"
Nov 03 22:32:58.116415 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:58.116343    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dde620e-e87f-4f87-984e-6358a642a100-istio-podinfo\") pod \"e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6\" (UID: \"7dde620e-e87f-4f87-984e-6358a642a100\") " pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6"
Nov 03 22:32:58.116515 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:58.116423    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dde620e-e87f-4f87-984e-6358a642a100-credential-socket\") pod \"e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6\" (UID: \"7dde620e-e87f-4f87-984e-6358a642a100\") " pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6"
Nov 03 22:32:58.117229 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:58.116920    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dde620e-e87f-4f87-984e-6358a642a100-workload-certs\") pod \"e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6\" (UID: \"7dde620e-e87f-4f87-984e-6358a642a100\") " pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6"
Nov 03 22:32:58.117504 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:58.117458    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dde620e-e87f-4f87-984e-6358a642a100-istio-data\") pod \"e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6\" (UID: \"7dde620e-e87f-4f87-984e-6358a642a100\") " pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6"
Nov 03 22:32:58.117649 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:58.117628    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dde620e-e87f-4f87-984e-6358a642a100-workload-socket\") pod \"e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6\" (UID: \"7dde620e-e87f-4f87-984e-6358a642a100\") " pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6"
Nov 03 22:32:58.117753 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:58.117732    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dde620e-e87f-4f87-984e-6358a642a100-credential-socket\") pod \"e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6\" (UID: \"7dde620e-e87f-4f87-984e-6358a642a100\") " pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6"
Nov 03 22:32:58.118044 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:58.118025    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dde620e-e87f-4f87-984e-6358a642a100-istiod-ca-cert\") pod \"e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6\" (UID: \"7dde620e-e87f-4f87-984e-6358a642a100\") " pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6"
Nov 03 22:32:58.119840 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:58.119812    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dde620e-e87f-4f87-984e-6358a642a100-istio-envoy\") pod \"e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6\" (UID: \"7dde620e-e87f-4f87-984e-6358a642a100\") " pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6"
Nov 03 22:32:58.120306 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:58.120286    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dde620e-e87f-4f87-984e-6358a642a100-istio-podinfo\") pod \"e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6\" (UID: \"7dde620e-e87f-4f87-984e-6358a642a100\") " pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6"
Nov 03 22:32:58.189999 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:58.162963    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdt8m\" (UniqueName: \"kubernetes.io/projected/7dde620e-e87f-4f87-984e-6358a642a100-kube-api-access-wdt8m\") pod \"e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6\" (UID: \"7dde620e-e87f-4f87-984e-6358a642a100\") " pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6"
Nov 03 22:32:58.189999 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:58.178902    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dde620e-e87f-4f87-984e-6358a642a100-istio-token\") pod \"e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6\" (UID: \"7dde620e-e87f-4f87-984e-6358a642a100\") " pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6"
Nov 03 22:32:58.221372 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:58.220104    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6"
Nov 03 22:32:58.693603 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:32:58.691136    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dde620e_e87f_4f87_984e_6358a642a100.slice/crio-40d49840b1c2132a88411cbedfe4c1648d0fc2b675edce31e27c40156e77b4be WatchSource:0}: Error finding container 40d49840b1c2132a88411cbedfe4c1648d0fc2b675edce31e27c40156e77b4be: Status 404 returned error can't find the container with id 40d49840b1c2132a88411cbedfe4c1648d0fc2b675edce31e27c40156e77b4be
Nov 03 22:32:58.697177 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:58.696539    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6"]
Nov 03 22:32:58.697418 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:58.697382    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:32:58.697509 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:58.697466    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:32:58.697569 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:58.697517    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:32:58.703838 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:58.703812    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:32:58.746248 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:58.736199    4431 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Nov 03 22:32:58.746248 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:58.736414    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq" podUID="a49eb997-50b7-45cf-a401-19c1a2b7af77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Nov 03 22:32:58.890241 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:58.879631    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:32:59.094301 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:59.093612    4431 patch_prober.go:29] interesting pod/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Nov 03 22:32:59.094301 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:59.093672    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7" podUID="6634b1a8-4b77-4281-8566-3470ecf3022f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Nov 03 22:32:59.104516 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:59.102834    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:32:59.114264 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:59.110555    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6" event={"ID":"7dde620e-e87f-4f87-984e-6358a642a100","Type":"ContainerStarted","Data":"75d3ccc913873d606bba13bb7360634f057d48dfdd96551ebf4e2803cc8981a2"}
Nov 03 22:32:59.114264 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:59.110611    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6" event={"ID":"7dde620e-e87f-4f87-984e-6358a642a100","Type":"ContainerStarted","Data":"40d49840b1c2132a88411cbedfe4c1648d0fc2b675edce31e27c40156e77b4be"}
Nov 03 22:32:59.173244 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:59.169543    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6" podStartSLOduration=2.169522169 podStartE2EDuration="2.169522169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:32:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:32:59.158237193 +0000 UTC m=+1203.184093056" watchObservedRunningTime="2026-11-03 22:32:59.169522169 +0000 UTC m=+1203.195378031"
Nov 03 22:32:59.221292 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:59.221255    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6"
Nov 03 22:32:59.222632 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:59.222605    4431 patch_prober.go:29] interesting pod/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Nov 03 22:32:59.223061 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:59.222659    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6" podUID="7dde620e-e87f-4f87-984e-6358a642a100" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Nov 03 22:32:59.736940 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:59.736908    4431 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Nov 03 22:32:59.737116 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:32:59.736955    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq" podUID="a49eb997-50b7-45cf-a401-19c1a2b7af77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Nov 03 22:33:00.108301 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:00.107343    4431 patch_prober.go:29] interesting pod/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Nov 03 22:33:00.108301 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:00.107413    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7" podUID="6634b1a8-4b77-4281-8566-3470ecf3022f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Nov 03 22:33:00.224716 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:00.221682    4431 patch_prober.go:29] interesting pod/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Nov 03 22:33:00.224716 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:00.221770    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6" podUID="7dde620e-e87f-4f87-984e-6358a642a100" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Nov 03 22:33:00.736526 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:00.736487    4431 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Nov 03 22:33:00.736694 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:00.736545    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq" podUID="a49eb997-50b7-45cf-a401-19c1a2b7af77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Nov 03 22:33:00.932867 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:00.932833    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:33:01.102344 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:01.096163    4431 patch_prober.go:29] interesting pod/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Nov 03 22:33:01.102344 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:01.096249    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7" podUID="6634b1a8-4b77-4281-8566-3470ecf3022f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Nov 03 22:33:01.134554 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:01.133130    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:33:01.210237 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:01.203493    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-2a2eca-6cf9b6f4bd-9mxw7"]
Nov 03 22:33:01.316243 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:01.306624    4431 patch_prober.go:29] interesting pod/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Nov 03 22:33:01.316243 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:01.306684    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6" podUID="7dde620e-e87f-4f87-984e-6358a642a100" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Nov 03 22:33:01.382030 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:01.381815    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:33:01.504370 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:01.504336    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-2a2eca-6cf9b6f4bd-9mxw7"]
Nov 03 22:33:01.510240 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:01.504694    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-2a2eca-6cf9b6f4bd-9mxw7"
Nov 03 22:33:01.595199 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:01.590776    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w6-2a2eca\""
Nov 03 22:33:01.595199 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:01.591371    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w6-2a2eca-dockercfg-9lnlz\""
Nov 03 22:33:01.764587 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:01.762883    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg8tz\" (UniqueName: \"kubernetes.io/projected/c9745b64-3b13-46c2-bbd3-68730b829f6d-kube-api-access-dg8tz\") pod \"payload-processing-e2e-shared-a-w6-2a2eca-6cf9b6f4bd-9mxw7\" (UID: \"c9745b64-3b13-46c2-bbd3-68730b829f6d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-2a2eca-6cf9b6f4bd-9mxw7"
Nov 03 22:33:01.764587 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:01.763101    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9745b64-3b13-46c2-bbd3-68730b829f6d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-2a2eca-6cf9b6f4bd-9mxw7\" (UID: \"c9745b64-3b13-46c2-bbd3-68730b829f6d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-2a2eca-6cf9b6f4bd-9mxw7"
Nov 03 22:33:01.823244 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:01.804357    4431 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Nov 03 22:33:01.823244 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:01.804420    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq" podUID="a49eb997-50b7-45cf-a401-19c1a2b7af77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Nov 03 22:33:01.905480 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:01.905327    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg8tz\" (UniqueName: \"kubernetes.io/projected/c9745b64-3b13-46c2-bbd3-68730b829f6d-kube-api-access-dg8tz\") pod \"payload-processing-e2e-shared-a-w6-2a2eca-6cf9b6f4bd-9mxw7\" (UID: \"c9745b64-3b13-46c2-bbd3-68730b829f6d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-2a2eca-6cf9b6f4bd-9mxw7"
Nov 03 22:33:01.906522 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:01.906495    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9745b64-3b13-46c2-bbd3-68730b829f6d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-2a2eca-6cf9b6f4bd-9mxw7\" (UID: \"c9745b64-3b13-46c2-bbd3-68730b829f6d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-2a2eca-6cf9b6f4bd-9mxw7"
Nov 03 22:33:01.933264 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:33:01.932558    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:33:01.933448 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:01.933267    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9745b64-3b13-46c2-bbd3-68730b829f6d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-2a2eca-6cf9b6f4bd-9mxw7\" (UID: \"c9745b64-3b13-46c2-bbd3-68730b829f6d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-2a2eca-6cf9b6f4bd-9mxw7"
Nov 03 22:33:01.987451 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:01.986876    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg8tz\" (UniqueName: \"kubernetes.io/projected/c9745b64-3b13-46c2-bbd3-68730b829f6d-kube-api-access-dg8tz\") pod \"payload-processing-e2e-shared-a-w6-2a2eca-6cf9b6f4bd-9mxw7\" (UID: \"c9745b64-3b13-46c2-bbd3-68730b829f6d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-2a2eca-6cf9b6f4bd-9mxw7"
Nov 03 22:33:02.033726 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:02.033636    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-2a2eca-6cf9b6f4bd-9mxw7"
Nov 03 22:33:02.104896 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:02.103030    4431 patch_prober.go:29] interesting pod/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Nov 03 22:33:02.104896 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:02.103093    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7" podUID="6634b1a8-4b77-4281-8566-3470ecf3022f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Nov 03 22:33:02.223960 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:02.221977    4431 patch_prober.go:29] interesting pod/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Nov 03 22:33:02.223960 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:02.222051    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6" podUID="7dde620e-e87f-4f87-984e-6358a642a100" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Nov 03 22:33:02.511917 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:02.511883    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-2a2eca-6cf9b6f4bd-9mxw7"]
Nov 03 22:33:02.512584 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:33:02.512561    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9745b64_3b13_46c2_bbd3_68730b829f6d.slice/crio-b696cbb1d9e2d6db5d6347ffb91accc92f8b71ef3886337bef2a88882b56e955 WatchSource:0}: Error finding container b696cbb1d9e2d6db5d6347ffb91accc92f8b71ef3886337bef2a88882b56e955: Status 404 returned error can't find the container with id b696cbb1d9e2d6db5d6347ffb91accc92f8b71ef3886337bef2a88882b56e955
Nov 03 22:33:02.736596 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:02.736557    4431 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Nov 03 22:33:02.736758 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:02.736625    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq" podUID="a49eb997-50b7-45cf-a401-19c1a2b7af77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Nov 03 22:33:03.097624 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:03.097596    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7"
Nov 03 22:33:03.098106 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:03.098074    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7"
Nov 03 22:33:03.098789 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:03.098767    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7"
Nov 03 22:33:03.220907 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:03.220867    4431 patch_prober.go:29] interesting pod/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Nov 03 22:33:03.221058 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:03.220929    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6" podUID="7dde620e-e87f-4f87-984e-6358a642a100" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Nov 03 22:33:03.472750 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:03.472712    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-2a2eca-6cf9b6f4bd-9mxw7" event={"ID":"c9745b64-3b13-46c2-bbd3-68730b829f6d","Type":"ContainerStarted","Data":"16b7139f8bf0c6b78969042463a948e1db54a750013a7d092a6f871d4fa0bbf4"}
Nov 03 22:33:03.472932 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:03.472772    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-2a2eca-6cf9b6f4bd-9mxw7" event={"ID":"c9745b64-3b13-46c2-bbd3-68730b829f6d","Type":"ContainerStarted","Data":"b696cbb1d9e2d6db5d6347ffb91accc92f8b71ef3886337bef2a88882b56e955"}
Nov 03 22:33:03.472997 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:03.472956    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-2a2eca-6cf9b6f4bd-9mxw7"
Nov 03 22:33:03.515395 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:03.515332    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-2a2eca-6cf9b6f4bd-9mxw7" podStartSLOduration=2.515312661 podStartE2EDuration="2.515312661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:33:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:33:03.513736049 +0000 UTC m=+1207.539591911" watchObservedRunningTime="2026-11-03 22:33:03.515312661 +0000 UTC m=+1207.541168523"
Nov 03 22:33:03.518573 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:03.518540    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:33:03.761298 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:03.751572    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq"
Nov 03 22:33:03.761298 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:03.753317    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq"
Nov 03 22:33:03.761298 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:03.758428    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq"
Nov 03 22:33:04.014240 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:04.014138    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-23d281-8599fb6555-qtrjc"
Nov 03 22:33:04.014397 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:04.014332    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-23d281-97b8bbd66-7dgnl"
Nov 03 22:33:04.220674 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:04.220638    4431 patch_prober.go:29] interesting pod/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Nov 03 22:33:04.220863 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:04.220704    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6" podUID="7dde620e-e87f-4f87-984e-6358a642a100" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Nov 03 22:33:05.220465 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:05.220427    4431 patch_prober.go:29] interesting pod/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Nov 03 22:33:05.220953 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:05.220486    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6" podUID="7dde620e-e87f-4f87-984e-6358a642a100" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Nov 03 22:33:05.265569 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:05.264811    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:33:05.933542 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:05.933510    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:33:06.225695 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:06.225610    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6"
Nov 03 22:33:06.226143 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:06.226098    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6"
Nov 03 22:33:06.226950 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:06.226931    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6"
Nov 03 22:33:06.892300 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:06.892266    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:33:07.088728 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:07.088701    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-8b1efeb6-5d7dd8ff94-q446n"
Nov 03 22:33:07.089167 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:07.089146    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-8b1efeb6-68787c86c9-wpjjk"
Nov 03 22:33:07.970343 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:07.970307    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:33:08.502472 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:08.502439    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:33:09.993364 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:09.993329    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:33:11.492155 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:11.492114    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:33:11.660387 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:11.660337    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:33:11.806100 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:11.806020    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:33:12.334469 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:12.334437    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:33:13.197179 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:13.197126    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:33:14.482895 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:14.482869    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-2a2eca-6cf9b6f4bd-9mxw7"
Nov 03 22:33:14.860889 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:14.860809    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:33:15.017002 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:15.016827    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:33:16.695013 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:16.694981    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:33:18.392123 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:18.392085    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:33:20.013736 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:20.013705    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:33:21.764615 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:21.760733    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:33:23.586239 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:23.571239    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:33:23.776239 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:23.767863    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:33:24.424243 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:24.424186    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:33:25.937843 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:25.937802    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:33:27.655476 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:27.655438    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:33:27.861683 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:27.861650    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:33:28.009184 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:28.009153    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:33:29.195460 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:29.195412    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:33:30.828094 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:30.828052    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:33:31.263773 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:33:31.263738    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:33:32.316772 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:32.316736    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:33:33.929227 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:33.929173    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:33:35.642194 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:35.642159    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:33:37.144288 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:37.144203    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:33:38.751765 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:38.751728    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:33:40.444503 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:40.444469    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:33:40.590941 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:40.590908    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:33:40.636654 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:40.636619    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz"]
Nov 03 22:33:40.642858 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:40.642831    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz"
Nov 03 22:33:40.646945 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:40.646901    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/892720cc-753b-43d6-ad3b-0494dff25623-istio-envoy\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz\" (UID: \"892720cc-753b-43d6-ad3b-0494dff25623\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz"
Nov 03 22:33:40.647095 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:40.646948    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/892720cc-753b-43d6-ad3b-0494dff25623-workload-certs\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz\" (UID: \"892720cc-753b-43d6-ad3b-0494dff25623\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz"
Nov 03 22:33:40.647181 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:40.647163    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/892720cc-753b-43d6-ad3b-0494dff25623-istio-token\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz\" (UID: \"892720cc-753b-43d6-ad3b-0494dff25623\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz"
Nov 03 22:33:40.647242 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:40.647192    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/892720cc-753b-43d6-ad3b-0494dff25623-istiod-ca-cert\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz\" (UID: \"892720cc-753b-43d6-ad3b-0494dff25623\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz"
Nov 03 22:33:40.647242 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:40.647223    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/892720cc-753b-43d6-ad3b-0494dff25623-istio-podinfo\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz\" (UID: \"892720cc-753b-43d6-ad3b-0494dff25623\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz"
Nov 03 22:33:40.647400 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:40.647376    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/892720cc-753b-43d6-ad3b-0494dff25623-workload-socket\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz\" (UID: \"892720cc-753b-43d6-ad3b-0494dff25623\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz"
Nov 03 22:33:40.647530 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:40.647507    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/892720cc-753b-43d6-ad3b-0494dff25623-credential-socket\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz\" (UID: \"892720cc-753b-43d6-ad3b-0494dff25623\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz"
Nov 03 22:33:40.647643 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:40.647564    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrq5p\" (UniqueName: \"kubernetes.io/projected/892720cc-753b-43d6-ad3b-0494dff25623-kube-api-access-vrq5p\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz\" (UID: \"892720cc-753b-43d6-ad3b-0494dff25623\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz"
Nov 03 22:33:40.647643 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:40.647604    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/892720cc-753b-43d6-ad3b-0494dff25623-istio-data\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz\" (UID: \"892720cc-753b-43d6-ad3b-0494dff25623\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz"
Nov 03 22:33:40.673036 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:40.673000    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz"]
Nov 03 22:33:40.748939 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:40.748845    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/892720cc-753b-43d6-ad3b-0494dff25623-istio-token\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz\" (UID: \"892720cc-753b-43d6-ad3b-0494dff25623\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz"
Nov 03 22:33:40.748939 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:40.748885    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/892720cc-753b-43d6-ad3b-0494dff25623-istiod-ca-cert\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz\" (UID: \"892720cc-753b-43d6-ad3b-0494dff25623\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz"
Nov 03 22:33:40.748939 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:40.748918    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/892720cc-753b-43d6-ad3b-0494dff25623-istio-podinfo\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz\" (UID: \"892720cc-753b-43d6-ad3b-0494dff25623\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz"
Nov 03 22:33:40.749261 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:40.748972    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/892720cc-753b-43d6-ad3b-0494dff25623-workload-socket\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz\" (UID: \"892720cc-753b-43d6-ad3b-0494dff25623\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz"
Nov 03 22:33:40.749261 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:40.749049    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/892720cc-753b-43d6-ad3b-0494dff25623-credential-socket\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz\" (UID: \"892720cc-753b-43d6-ad3b-0494dff25623\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz"
Nov 03 22:33:40.749261 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:40.749078    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrq5p\" (UniqueName: \"kubernetes.io/projected/892720cc-753b-43d6-ad3b-0494dff25623-kube-api-access-vrq5p\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz\" (UID: \"892720cc-753b-43d6-ad3b-0494dff25623\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz"
Nov 03 22:33:40.749261 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:40.749108    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/892720cc-753b-43d6-ad3b-0494dff25623-istio-data\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz\" (UID: \"892720cc-753b-43d6-ad3b-0494dff25623\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz"
Nov 03 22:33:40.749261 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:40.749247    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/892720cc-753b-43d6-ad3b-0494dff25623-istio-envoy\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz\" (UID: \"892720cc-753b-43d6-ad3b-0494dff25623\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz"
Nov 03 22:33:40.749539 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:40.749291    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/892720cc-753b-43d6-ad3b-0494dff25623-workload-certs\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz\" (UID: \"892720cc-753b-43d6-ad3b-0494dff25623\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz"
Nov 03 22:33:40.749754 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:40.749730    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/892720cc-753b-43d6-ad3b-0494dff25623-istiod-ca-cert\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz\" (UID: \"892720cc-753b-43d6-ad3b-0494dff25623\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz"
Nov 03 22:33:40.750046 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:40.750028    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/892720cc-753b-43d6-ad3b-0494dff25623-workload-socket\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz\" (UID: \"892720cc-753b-43d6-ad3b-0494dff25623\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz"
Nov 03 22:33:40.750186 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:40.750162    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/892720cc-753b-43d6-ad3b-0494dff25623-workload-certs\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz\" (UID: \"892720cc-753b-43d6-ad3b-0494dff25623\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz"
Nov 03 22:33:40.750309 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:40.750286    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/892720cc-753b-43d6-ad3b-0494dff25623-credential-socket\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz\" (UID: \"892720cc-753b-43d6-ad3b-0494dff25623\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz"
Nov 03 22:33:40.750794 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:40.750767    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/892720cc-753b-43d6-ad3b-0494dff25623-istio-data\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz\" (UID: \"892720cc-753b-43d6-ad3b-0494dff25623\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz"
Nov 03 22:33:40.752070 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:40.752040    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/892720cc-753b-43d6-ad3b-0494dff25623-istio-envoy\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz\" (UID: \"892720cc-753b-43d6-ad3b-0494dff25623\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz"
Nov 03 22:33:40.752692 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:40.752668    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/892720cc-753b-43d6-ad3b-0494dff25623-istio-podinfo\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz\" (UID: \"892720cc-753b-43d6-ad3b-0494dff25623\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz"
Nov 03 22:33:40.778385 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:40.778354    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrq5p\" (UniqueName: \"kubernetes.io/projected/892720cc-753b-43d6-ad3b-0494dff25623-kube-api-access-vrq5p\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz\" (UID: \"892720cc-753b-43d6-ad3b-0494dff25623\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz"
Nov 03 22:33:40.778573 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:40.778549    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/892720cc-753b-43d6-ad3b-0494dff25623-istio-token\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz\" (UID: \"892720cc-753b-43d6-ad3b-0494dff25623\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz"
Nov 03 22:33:40.966933 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:40.966896    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz"
Nov 03 22:33:41.058258 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:41.058125    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:33:41.296741 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:41.296709    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz"]
Nov 03 22:33:41.297044 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:33:41.297022    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod892720cc_753b_43d6_ad3b_0494dff25623.slice/crio-b8cc2f784af8f2c518c5d4e8648a01dd71e6503f84863a6bb52ffd6933e5701b WatchSource:0}: Error finding container b8cc2f784af8f2c518c5d4e8648a01dd71e6503f84863a6bb52ffd6933e5701b: Status 404 returned error can't find the container with id b8cc2f784af8f2c518c5d4e8648a01dd71e6503f84863a6bb52ffd6933e5701b
Nov 03 22:33:41.299749 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:41.299718    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:33:41.299825 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:41.299787    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:33:41.299869 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:41.299828    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:33:41.564916 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:41.564823    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:33:41.729146 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:41.729111    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:33:42.063239 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:42.062797    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz" event={"ID":"892720cc-753b-43d6-ad3b-0494dff25623","Type":"ContainerStarted","Data":"e863a9a320c2c2afdc4ebbb5f294c33044671e9cc1341972243229c285e3190b"}
Nov 03 22:33:42.063239 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:42.062864    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz" event={"ID":"892720cc-753b-43d6-ad3b-0494dff25623","Type":"ContainerStarted","Data":"b8cc2f784af8f2c518c5d4e8648a01dd71e6503f84863a6bb52ffd6933e5701b"}
Nov 03 22:33:42.174479 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:42.174301    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz" podStartSLOduration=2.174279445 podStartE2EDuration="2.174279445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:33:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:33:42.159866448 +0000 UTC m=+1246.185722317" watchObservedRunningTime="2026-11-03 22:33:42.174279445 +0000 UTC m=+1246.200135311"
Nov 03 22:33:42.296239 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:42.295816    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:33:42.605548 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:42.605478    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:33:42.769244 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:42.768603    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:33:42.969860 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:42.967574    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz"
Nov 03 22:33:42.970190 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:42.970167    4431 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Nov 03 22:33:42.970395 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:42.970370    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz" podUID="892720cc-753b-43d6-ad3b-0494dff25623" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Nov 03 22:33:43.380982 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:43.380886    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:33:43.967890 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:43.967854    4431 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Nov 03 22:33:43.968358 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:43.967909    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz" podUID="892720cc-753b-43d6-ad3b-0494dff25623" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Nov 03 22:33:44.218416 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:44.218325    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:33:44.969749 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:44.969704    4431 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Nov 03 22:33:44.970339 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:44.969776    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz" podUID="892720cc-753b-43d6-ad3b-0494dff25623" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Nov 03 22:33:45.968064 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:45.968020    4431 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Nov 03 22:33:45.968297 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:45.968075    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz" podUID="892720cc-753b-43d6-ad3b-0494dff25623" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Nov 03 22:33:45.980834 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:45.980789    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:33:46.331695 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:46.331621    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:33:46.498189 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:46.498158    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:33:46.665296 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:46.659781    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:33:46.817933 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:46.817900    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:33:46.967756 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:46.967676    4431 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Nov 03 22:33:46.967966 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:46.967945    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz" podUID="892720cc-753b-43d6-ad3b-0494dff25623" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Nov 03 22:33:47.901324 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:47.901286    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:33:47.969236 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:47.967865    4431 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Nov 03 22:33:47.969236 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:47.967947    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz" podUID="892720cc-753b-43d6-ad3b-0494dff25623" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Nov 03 22:33:48.139444 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:48.130140    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:33:48.291973 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:48.291935    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-d8a3d1-85f887b8c4-gfs5q"]
Nov 03 22:33:48.295823 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:48.295787    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:33:48.298533 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:48.298506    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d8a3d1-85f887b8c4-gfs5q"
Nov 03 22:33:48.322373 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:48.322341    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-d8a3d1-85f887b8c4-gfs5q"]
Nov 03 22:33:48.465248 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:48.458704    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-d8a3d1-dockercfg-s9kjb\""
Nov 03 22:33:48.562375 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:48.499899    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-d8a3d1\""
Nov 03 22:33:48.562375 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:48.545588    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqw9v\" (UniqueName: \"kubernetes.io/projected/94198afd-2296-4658-ba19-6624a26f2bf9-kube-api-access-zqw9v\") pod \"payload-processing-e2e-shared-b-w3-d8a3d1-85f887b8c4-gfs5q\" (UID: \"94198afd-2296-4658-ba19-6624a26f2bf9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d8a3d1-85f887b8c4-gfs5q"
Nov 03 22:33:48.562375 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:48.545716    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94198afd-2296-4658-ba19-6624a26f2bf9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-d8a3d1-85f887b8c4-gfs5q\" (UID: \"94198afd-2296-4658-ba19-6624a26f2bf9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d8a3d1-85f887b8c4-gfs5q"
Nov 03 22:33:48.647806 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:48.647762    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqw9v\" (UniqueName: \"kubernetes.io/projected/94198afd-2296-4658-ba19-6624a26f2bf9-kube-api-access-zqw9v\") pod \"payload-processing-e2e-shared-b-w3-d8a3d1-85f887b8c4-gfs5q\" (UID: \"94198afd-2296-4658-ba19-6624a26f2bf9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d8a3d1-85f887b8c4-gfs5q"
Nov 03 22:33:48.648167 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:48.648146    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94198afd-2296-4658-ba19-6624a26f2bf9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-d8a3d1-85f887b8c4-gfs5q\" (UID: \"94198afd-2296-4658-ba19-6624a26f2bf9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d8a3d1-85f887b8c4-gfs5q"
Nov 03 22:33:48.649039 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:48.649013    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94198afd-2296-4658-ba19-6624a26f2bf9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-d8a3d1-85f887b8c4-gfs5q\" (UID: \"94198afd-2296-4658-ba19-6624a26f2bf9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d8a3d1-85f887b8c4-gfs5q"
Nov 03 22:33:48.673312 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:48.673199    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqw9v\" (UniqueName: \"kubernetes.io/projected/94198afd-2296-4658-ba19-6624a26f2bf9-kube-api-access-zqw9v\") pod \"payload-processing-e2e-shared-b-w3-d8a3d1-85f887b8c4-gfs5q\" (UID: \"94198afd-2296-4658-ba19-6624a26f2bf9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d8a3d1-85f887b8c4-gfs5q"
Nov 03 22:33:48.792549 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:48.792513    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:33:48.933346 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:48.933313    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d8a3d1-85f887b8c4-gfs5q"
Nov 03 22:33:48.976588 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:48.976530    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz"
Nov 03 22:33:48.977945 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:48.977923    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz"
Nov 03 22:33:48.978857 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:48.978801    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz"
Nov 03 22:33:49.109296 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:49.108006    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:33:49.251598 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:49.251507    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb"]
Nov 03 22:33:49.258554 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:49.258512    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb"
Nov 03 22:33:49.304926 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:49.303900    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-4c5345-openshift-default-dockercfg-7q5wm\""
Nov 03 22:33:49.305930 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:49.305883    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb"]
Nov 03 22:33:49.471558 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:49.471514    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:33:49.540164 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:49.533644    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f44ff83c-d054-4f20-a04c-c6204de2bae2-workload-certs\") pod \"e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb\" (UID: \"f44ff83c-d054-4f20-a04c-c6204de2bae2\") " pod="openshift-ingress/e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb"
Nov 03 22:33:49.540164 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:49.533749    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f44ff83c-d054-4f20-a04c-c6204de2bae2-credential-socket\") pod \"e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb\" (UID: \"f44ff83c-d054-4f20-a04c-c6204de2bae2\") " pod="openshift-ingress/e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb"
Nov 03 22:33:49.540164 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:49.533919    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f44ff83c-d054-4f20-a04c-c6204de2bae2-istio-envoy\") pod \"e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb\" (UID: \"f44ff83c-d054-4f20-a04c-c6204de2bae2\") " pod="openshift-ingress/e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb"
Nov 03 22:33:49.540164 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:49.533994    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcf58\" (UniqueName: \"kubernetes.io/projected/f44ff83c-d054-4f20-a04c-c6204de2bae2-kube-api-access-kcf58\") pod \"e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb\" (UID: \"f44ff83c-d054-4f20-a04c-c6204de2bae2\") " pod="openshift-ingress/e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb"
Nov 03 22:33:49.540164 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:49.534119    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f44ff83c-d054-4f20-a04c-c6204de2bae2-istio-data\") pod \"e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb\" (UID: \"f44ff83c-d054-4f20-a04c-c6204de2bae2\") " pod="openshift-ingress/e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb"
Nov 03 22:33:49.540164 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:49.534190    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f44ff83c-d054-4f20-a04c-c6204de2bae2-istio-podinfo\") pod \"e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb\" (UID: \"f44ff83c-d054-4f20-a04c-c6204de2bae2\") " pod="openshift-ingress/e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb"
Nov 03 22:33:49.540164 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:49.534276    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f44ff83c-d054-4f20-a04c-c6204de2bae2-istio-token\") pod \"e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb\" (UID: \"f44ff83c-d054-4f20-a04c-c6204de2bae2\") " pod="openshift-ingress/e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb"
Nov 03 22:33:49.655236 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:49.653375    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f44ff83c-d054-4f20-a04c-c6204de2bae2-istiod-ca-cert\") pod \"e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb\" (UID: \"f44ff83c-d054-4f20-a04c-c6204de2bae2\") " pod="openshift-ingress/e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb"
Nov 03 22:33:49.655236 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:49.653679    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f44ff83c-d054-4f20-a04c-c6204de2bae2-workload-socket\") pod \"e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb\" (UID: \"f44ff83c-d054-4f20-a04c-c6204de2bae2\") " pod="openshift-ingress/e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb"
Nov 03 22:33:49.655236 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:49.645430    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:33:49.752436 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:49.752406    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-d8a3d1-85f887b8c4-gfs5q"]
Nov 03 22:33:49.765716 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:49.758135    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f44ff83c-d054-4f20-a04c-c6204de2bae2-istio-data\") pod \"e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb\" (UID: \"f44ff83c-d054-4f20-a04c-c6204de2bae2\") " pod="openshift-ingress/e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb"
Nov 03 22:33:49.765716 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:49.758238    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f44ff83c-d054-4f20-a04c-c6204de2bae2-istio-podinfo\") pod \"e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb\" (UID: \"f44ff83c-d054-4f20-a04c-c6204de2bae2\") " pod="openshift-ingress/e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb"
Nov 03 22:33:49.765716 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:49.758306    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f44ff83c-d054-4f20-a04c-c6204de2bae2-istio-token\") pod \"e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb\" (UID: \"f44ff83c-d054-4f20-a04c-c6204de2bae2\") " pod="openshift-ingress/e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb"
Nov 03 22:33:49.765716 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:49.758330    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f44ff83c-d054-4f20-a04c-c6204de2bae2-istiod-ca-cert\") pod \"e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb\" (UID: \"f44ff83c-d054-4f20-a04c-c6204de2bae2\") " pod="openshift-ingress/e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb"
Nov 03 22:33:49.765716 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:49.758382    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f44ff83c-d054-4f20-a04c-c6204de2bae2-workload-socket\") pod \"e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb\" (UID: \"f44ff83c-d054-4f20-a04c-c6204de2bae2\") " pod="openshift-ingress/e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb"
Nov 03 22:33:49.765716 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:49.758435    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f44ff83c-d054-4f20-a04c-c6204de2bae2-workload-certs\") pod \"e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb\" (UID: \"f44ff83c-d054-4f20-a04c-c6204de2bae2\") " pod="openshift-ingress/e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb"
Nov 03 22:33:49.765716 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:49.758510    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f44ff83c-d054-4f20-a04c-c6204de2bae2-credential-socket\") pod \"e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb\" (UID: \"f44ff83c-d054-4f20-a04c-c6204de2bae2\") " pod="openshift-ingress/e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb"
Nov 03 22:33:49.765716 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:49.758665    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f44ff83c-d054-4f20-a04c-c6204de2bae2-istio-envoy\") pod \"e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb\" (UID: \"f44ff83c-d054-4f20-a04c-c6204de2bae2\") " pod="openshift-ingress/e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb"
Nov 03 22:33:49.765716 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:49.758732    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcf58\" (UniqueName: \"kubernetes.io/projected/f44ff83c-d054-4f20-a04c-c6204de2bae2-kube-api-access-kcf58\") pod \"e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb\" (UID: \"f44ff83c-d054-4f20-a04c-c6204de2bae2\") " pod="openshift-ingress/e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb"
Nov 03 22:33:49.765716 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:49.758874    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f44ff83c-d054-4f20-a04c-c6204de2bae2-istio-data\") pod \"e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb\" (UID: \"f44ff83c-d054-4f20-a04c-c6204de2bae2\") " pod="openshift-ingress/e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb"
Nov 03 22:33:49.765716 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:49.759783    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f44ff83c-d054-4f20-a04c-c6204de2bae2-istiod-ca-cert\") pod \"e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb\" (UID: \"f44ff83c-d054-4f20-a04c-c6204de2bae2\") " pod="openshift-ingress/e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb"
Nov 03 22:33:49.765716 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:49.759962    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f44ff83c-d054-4f20-a04c-c6204de2bae2-workload-socket\") pod \"e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb\" (UID: \"f44ff83c-d054-4f20-a04c-c6204de2bae2\") " pod="openshift-ingress/e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb"
Nov 03 22:33:49.765716 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:49.760069    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f44ff83c-d054-4f20-a04c-c6204de2bae2-credential-socket\") pod \"e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb\" (UID: \"f44ff83c-d054-4f20-a04c-c6204de2bae2\") " pod="openshift-ingress/e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb"
Nov 03 22:33:49.778003 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:49.777964    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f44ff83c-d054-4f20-a04c-c6204de2bae2-workload-certs\") pod \"e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb\" (UID: \"f44ff83c-d054-4f20-a04c-c6204de2bae2\") " pod="openshift-ingress/e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb"
Nov 03 22:33:49.782811 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:49.782754    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f44ff83c-d054-4f20-a04c-c6204de2bae2-istio-envoy\") pod \"e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb\" (UID: \"f44ff83c-d054-4f20-a04c-c6204de2bae2\") " pod="openshift-ingress/e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb"
Nov 03 22:33:49.786676 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:49.786536    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f44ff83c-d054-4f20-a04c-c6204de2bae2-istio-token\") pod \"e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb\" (UID: \"f44ff83c-d054-4f20-a04c-c6204de2bae2\") " pod="openshift-ingress/e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb"
Nov 03 22:33:49.788830 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:49.788801    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcf58\" (UniqueName: \"kubernetes.io/projected/f44ff83c-d054-4f20-a04c-c6204de2bae2-kube-api-access-kcf58\") pod \"e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb\" (UID: \"f44ff83c-d054-4f20-a04c-c6204de2bae2\") " pod="openshift-ingress/e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb"
Nov 03 22:33:49.789764 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:49.789722    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f44ff83c-d054-4f20-a04c-c6204de2bae2-istio-podinfo\") pod \"e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb\" (UID: \"f44ff83c-d054-4f20-a04c-c6204de2bae2\") " pod="openshift-ingress/e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb"
Nov 03 22:33:49.879825 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:49.879780    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb"
Nov 03 22:33:50.202536 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:50.202498    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d8a3d1-85f887b8c4-gfs5q" event={"ID":"94198afd-2296-4658-ba19-6624a26f2bf9","Type":"ContainerStarted","Data":"d3109ce13320cddcfdadb8985aa6bb60d472eb6cf56ed093b5485bb2799c820c"}
Nov 03 22:33:50.203023 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:50.202555    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d8a3d1-85f887b8c4-gfs5q" event={"ID":"94198afd-2296-4658-ba19-6624a26f2bf9","Type":"ContainerStarted","Data":"4e5d86cafbf9719ba74d778496bf3249f9dc4d459a7464f4ee7f4f6f1de91140"}
Nov 03 22:33:50.203023 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:50.202628    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d8a3d1-85f887b8c4-gfs5q"
Nov 03 22:33:50.229688 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:50.229659    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb"]
Nov 03 22:33:50.261543 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:33:50.261505    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf44ff83c_d054_4f20_a04c_c6204de2bae2.slice/crio-baf73975af674c319d2f697566bae2366c90d133227035695897b85952f23e97 WatchSource:0}: Error finding container baf73975af674c319d2f697566bae2366c90d133227035695897b85952f23e97: Status 404 returned error can't find the container with id baf73975af674c319d2f697566bae2366c90d133227035695897b85952f23e97
Nov 03 22:33:50.263663 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:50.263616    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d8a3d1-85f887b8c4-gfs5q" podStartSLOduration=2.263602809 podStartE2EDuration="2.263602809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:33:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:33:50.242847475 +0000 UTC m=+1254.268703364" watchObservedRunningTime="2026-11-03 22:33:50.263602809 +0000 UTC m=+1254.289458671"
Nov 03 22:33:50.264186 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:50.264152    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:33:50.264349 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:50.264296    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:33:50.264422 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:50.264349    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:33:50.272836 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:50.272797    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:33:51.215345 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:51.215273    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb" event={"ID":"f44ff83c-d054-4f20-a04c-c6204de2bae2","Type":"ContainerStarted","Data":"fd8992222dab0b30bb72530b1349c352047bd2118d3f30adca4cb7b736c3d0cb"}
Nov 03 22:33:51.215874 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:51.215405    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb" event={"ID":"f44ff83c-d054-4f20-a04c-c6204de2bae2","Type":"ContainerStarted","Data":"baf73975af674c319d2f697566bae2366c90d133227035695897b85952f23e97"}
Nov 03 22:33:51.275770 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:51.275712    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb" podStartSLOduration=2.275696071 podStartE2EDuration="2.275696071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:33:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:33:51.271316138 +0000 UTC m=+1255.297172000" watchObservedRunningTime="2026-11-03 22:33:51.275696071 +0000 UTC m=+1255.301551933"
Nov 03 22:33:51.557543 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:51.557436    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:33:51.880278 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:51.880181    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb"
Nov 03 22:33:51.888268 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:51.888197    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb"
Nov 03 22:33:51.908656 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:51.908619    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:33:52.224723 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:52.224689    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb"
Nov 03 22:33:52.225731 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:52.225705    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb"
Nov 03 22:33:53.509594 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:53.509560    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:33:53.667571 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:53.667521    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:33:53.810374 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:53.810280    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:33:54.657751 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:54.657710    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:33:55.120906 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:55.120870    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:33:55.329195 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:55.329156    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:33:55.498938 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:55.498902    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:33:55.501020 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:55.500992    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4c5345-5548f954c-kk85c"]
Nov 03 22:33:55.506304 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:55.506281    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4c5345-5548f954c-kk85c"
Nov 03 22:33:55.528651 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:55.528621    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4c5345-5548f954c-kk85c"]
Nov 03 22:33:55.533721 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:55.533552    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-4c5345\""
Nov 03 22:33:55.533861 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:55.533742    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-4c5345-dockercfg-4rpmw\""
Nov 03 22:33:55.630140 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:55.630093    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07bc79a3-84e3-4852-a5cb-6cd9c0d9eceb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-4c5345-5548f954c-kk85c\" (UID: \"07bc79a3-84e3-4852-a5cb-6cd9c0d9eceb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4c5345-5548f954c-kk85c"
Nov 03 22:33:55.630441 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:55.630418    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7c75\" (UniqueName: \"kubernetes.io/projected/07bc79a3-84e3-4852-a5cb-6cd9c0d9eceb-kube-api-access-g7c75\") pod \"payload-pre-processing-e2e-api-a-4c5345-5548f954c-kk85c\" (UID: \"07bc79a3-84e3-4852-a5cb-6cd9c0d9eceb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4c5345-5548f954c-kk85c"
Nov 03 22:33:55.658499 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:55.658457    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:33:55.732666 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:55.732618    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07bc79a3-84e3-4852-a5cb-6cd9c0d9eceb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-4c5345-5548f954c-kk85c\" (UID: \"07bc79a3-84e3-4852-a5cb-6cd9c0d9eceb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4c5345-5548f954c-kk85c"
Nov 03 22:33:55.754824 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:55.754732    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7c75\" (UniqueName: \"kubernetes.io/projected/07bc79a3-84e3-4852-a5cb-6cd9c0d9eceb-kube-api-access-g7c75\") pod \"payload-pre-processing-e2e-api-a-4c5345-5548f954c-kk85c\" (UID: \"07bc79a3-84e3-4852-a5cb-6cd9c0d9eceb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4c5345-5548f954c-kk85c"
Nov 03 22:33:55.755451 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:55.733838    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07bc79a3-84e3-4852-a5cb-6cd9c0d9eceb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-4c5345-5548f954c-kk85c\" (UID: \"07bc79a3-84e3-4852-a5cb-6cd9c0d9eceb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4c5345-5548f954c-kk85c"
Nov 03 22:33:55.808455 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:55.808412    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7c75\" (UniqueName: \"kubernetes.io/projected/07bc79a3-84e3-4852-a5cb-6cd9c0d9eceb-kube-api-access-g7c75\") pod \"payload-pre-processing-e2e-api-a-4c5345-5548f954c-kk85c\" (UID: \"07bc79a3-84e3-4852-a5cb-6cd9c0d9eceb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4c5345-5548f954c-kk85c"
Nov 03 22:33:55.847556 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:55.847517    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4c5345-5548f954c-kk85c"
Nov 03 22:33:56.229920 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:56.229870    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4c5345-5548f954c-kk85c"]
Nov 03 22:33:56.235998 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:33:56.235958    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07bc79a3_84e3_4852_a5cb_6cd9c0d9eceb.slice/crio-d70545080ff3579395ba09d30d23bb9a63d4642354397cb614832ed5de67ac7f WatchSource:0}: Error finding container d70545080ff3579395ba09d30d23bb9a63d4642354397cb614832ed5de67ac7f: Status 404 returned error can't find the container with id d70545080ff3579395ba09d30d23bb9a63d4642354397cb614832ed5de67ac7f
Nov 03 22:33:56.258989 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:56.258946    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4c5345-5548f954c-kk85c" event={"ID":"07bc79a3-84e3-4852-a5cb-6cd9c0d9eceb","Type":"ContainerStarted","Data":"d70545080ff3579395ba09d30d23bb9a63d4642354397cb614832ed5de67ac7f"}
Nov 03 22:33:56.549446 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:56.549338    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:33:56.941948 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:56.941909    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:33:57.300942 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:57.300848    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4c5345-5548f954c-kk85c" event={"ID":"07bc79a3-84e3-4852-a5cb-6cd9c0d9eceb","Type":"ContainerStarted","Data":"1d0fb38f1f0ff2fc1653daad57a782510ce80de9122f4a75d0eb2dffb16555fe"}
Nov 03 22:33:57.300942 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:57.300936    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4c5345-5548f954c-kk85c"
Nov 03 22:33:57.339973 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:57.339919    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4c5345-5548f954c-kk85c" podStartSLOduration=2.339902142 podStartE2EDuration="2.339902142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:33:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:33:57.339625257 +0000 UTC m=+1261.365481111" watchObservedRunningTime="2026-11-03 22:33:57.339902142 +0000 UTC m=+1261.365758004"
Nov 03 22:33:58.457424 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:58.457384    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:33:58.752819 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:58.752733    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:33:59.539537 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:59.539502    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:33:59.685209 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:33:59.685175    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:34:00.738245 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:34:00.737808    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:34:01.223293 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:34:01.222200    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d8a3d1-85f887b8c4-gfs5q"
Nov 03 22:34:01.263444 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:34:01.263404    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:34:01.799025 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:34:01.798985    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:34:01.951087 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:34:01.951051    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:34:02.752695 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:34:02.752632    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:34:04.689853 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:34:04.689818    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:34:04.904256 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:34:04.904179    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:34:06.713093 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:34:06.713054    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:34:08.312821 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:34:08.312791    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4c5345-5548f954c-kk85c"
Nov 03 22:34:08.586446 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:34:08.586370    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:34:09.631110 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:34:09.631069    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:34:10.293745 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:34:10.293705    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:34:11.846478 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:34:11.846429    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:34:13.304020 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:34:13.303978    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:34:13.449095 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:34:13.449022    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:34:13.591314 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:34:13.591192    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:34:15.084533 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:34:15.084473    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:34:16.703826 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:34:16.703789    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:34:16.997159 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:34:16.997070    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:34:18.406090 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:34:18.406048    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:34:20.426573 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:34:20.426509    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:34:22.226622 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:34:22.226584    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:34:22.381239 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:34:22.379280    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:34:24.008091 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:34:24.008052    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:34:25.589199 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:34:25.589166    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:34:27.215870 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:34:27.215834    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:34:28.772487 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:34:28.772452    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:34:28.913231 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:34:28.913162    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:34:30.420994 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:34:30.420952    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:34:31.265532 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:34:31.265492    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:34:32.054979 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:34:32.054939    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:34:33.670547 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:34:33.670482    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:34:35.387575 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:34:35.387535    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:34:37.122369 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:34:37.122315    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:34:38.845577 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:34:38.845536    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:34:40.403549 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:34:40.403454    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:34:42.093257 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:34:42.093187    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:34:43.612089 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:34:43.612054    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:34:45.541470 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:34:45.541427    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:34:46.675952 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:34:46.675920    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:34:46.968618 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:34:46.968532    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:34:47.336482 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:34:47.336353    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:34:47.526411 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:34:47.526375    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:34:48.782143 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:34:48.782050    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:34:49.135882 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:34:49.135775    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:34:49.566333 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:34:49.566292    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:34:49.722062 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:34:49.722021    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:34:50.748402 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:34:50.748361    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:34:51.700275 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:34:51.700207    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:34:52.272984 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:34:52.272953    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:34:53.750670 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:34:53.750635    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:34:53.894606 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:34:53.894572    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:34:54.060894 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:34:54.060768    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:34:54.442860 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:34:54.442825    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:34:54.866470 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:34:54.866361    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:34:55.572378 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:34:55.572328    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:34:55.893585 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:34:55.893482    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:34:56.082564 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:34:56.082532    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:34:56.288241 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:34:56.287329    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:34:57.824288 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:34:57.824251    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:34:59.480914 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:34:59.480874    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:35:00.675770 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:00.675737    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:35:00.844359 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:00.844324    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:35:01.299205 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:35:01.299164    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:35:01.560331 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:01.560230    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:35:03.206294 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:03.206210    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:35:03.387321 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:03.387290    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bbfc2e-6ddb5f6584-hc4kx"]
Nov 03 22:35:03.392153 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:03.392133    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-bbfc2e-6ddb5f6584-hc4kx"
Nov 03 22:35:03.400189 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:03.400162    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:35:03.410959 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:03.410931    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bbfc2e-6ddb5f6584-hc4kx"]
Nov 03 22:35:03.415742 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:03.415719    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-bbfc2e\""
Nov 03 22:35:03.450971 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:03.450935    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-bbfc2e-dockercfg-x6wkj\""
Nov 03 22:35:03.520239 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:03.514138    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crm7c\" (UniqueName: \"kubernetes.io/projected/7e7cb728-8ff5-431d-80d8-87cffee7f726-kube-api-access-crm7c\") pod \"payload-processing-e2e-api-b-bbfc2e-6ddb5f6584-hc4kx\" (UID: \"7e7cb728-8ff5-431d-80d8-87cffee7f726\") " pod="openshift-ingress/payload-processing-e2e-api-b-bbfc2e-6ddb5f6584-hc4kx"
Nov 03 22:35:03.520239 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:03.514544    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e7cb728-8ff5-431d-80d8-87cffee7f726-plugins-config-volume\") pod \"payload-processing-e2e-api-b-bbfc2e-6ddb5f6584-hc4kx\" (UID: \"7e7cb728-8ff5-431d-80d8-87cffee7f726\") " pod="openshift-ingress/payload-processing-e2e-api-b-bbfc2e-6ddb5f6584-hc4kx"
Nov 03 22:35:03.589425 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:03.589391    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:35:03.618271 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:03.618233    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e7cb728-8ff5-431d-80d8-87cffee7f726-plugins-config-volume\") pod \"payload-processing-e2e-api-b-bbfc2e-6ddb5f6584-hc4kx\" (UID: \"7e7cb728-8ff5-431d-80d8-87cffee7f726\") " pod="openshift-ingress/payload-processing-e2e-api-b-bbfc2e-6ddb5f6584-hc4kx"
Nov 03 22:35:03.618733 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:03.618612    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crm7c\" (UniqueName: \"kubernetes.io/projected/7e7cb728-8ff5-431d-80d8-87cffee7f726-kube-api-access-crm7c\") pod \"payload-processing-e2e-api-b-bbfc2e-6ddb5f6584-hc4kx\" (UID: \"7e7cb728-8ff5-431d-80d8-87cffee7f726\") " pod="openshift-ingress/payload-processing-e2e-api-b-bbfc2e-6ddb5f6584-hc4kx"
Nov 03 22:35:03.619686 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:03.619663    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e7cb728-8ff5-431d-80d8-87cffee7f726-plugins-config-volume\") pod \"payload-processing-e2e-api-b-bbfc2e-6ddb5f6584-hc4kx\" (UID: \"7e7cb728-8ff5-431d-80d8-87cffee7f726\") " pod="openshift-ingress/payload-processing-e2e-api-b-bbfc2e-6ddb5f6584-hc4kx"
Nov 03 22:35:03.642418 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:03.642389    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crm7c\" (UniqueName: \"kubernetes.io/projected/7e7cb728-8ff5-431d-80d8-87cffee7f726-kube-api-access-crm7c\") pod \"payload-processing-e2e-api-b-bbfc2e-6ddb5f6584-hc4kx\" (UID: \"7e7cb728-8ff5-431d-80d8-87cffee7f726\") " pod="openshift-ingress/payload-processing-e2e-api-b-bbfc2e-6ddb5f6584-hc4kx"
Nov 03 22:35:03.709371 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:03.709339    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-bbfc2e-6ddb5f6584-hc4kx"
Nov 03 22:35:04.054150 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:04.054116    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bbfc2e-6ddb5f6584-hc4kx"]
Nov 03 22:35:04.055380 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:35:04.055350    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e7cb728_8ff5_431d_80d8_87cffee7f726.slice/crio-68d3e20553b9724caa8daf15a37e284a7f7d73924a3c45ce7faae645bdb238b9 WatchSource:0}: Error finding container 68d3e20553b9724caa8daf15a37e284a7f7d73924a3c45ce7faae645bdb238b9: Status 404 returned error can't find the container with id 68d3e20553b9724caa8daf15a37e284a7f7d73924a3c45ce7faae645bdb238b9
Nov 03 22:35:04.985419 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:04.985381    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-bbfc2e-6ddb5f6584-hc4kx" event={"ID":"7e7cb728-8ff5-431d-80d8-87cffee7f726","Type":"ContainerStarted","Data":"14a52d98b83639dedc5ef62330b48a9c22cac48c01b8886ef0b8b8657d4b9121"}
Nov 03 22:35:04.985840 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:04.985452    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-bbfc2e-6ddb5f6584-hc4kx"
Nov 03 22:35:04.985840 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:04.985477    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-bbfc2e-6ddb5f6584-hc4kx" event={"ID":"7e7cb728-8ff5-431d-80d8-87cffee7f726","Type":"ContainerStarted","Data":"68d3e20553b9724caa8daf15a37e284a7f7d73924a3c45ce7faae645bdb238b9"}
Nov 03 22:35:05.029704 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:05.029654    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-bbfc2e-6ddb5f6584-hc4kx" podStartSLOduration=2.02964056 podStartE2EDuration="2.02964056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:35:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:35:05.02566543 +0000 UTC m=+1329.051521292" watchObservedRunningTime="2026-11-03 22:35:05.02964056 +0000 UTC m=+1329.055496422"
Nov 03 22:35:05.124088 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:05.124053    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:35:05.838597 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:05.819449    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:35:06.781549 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:06.781515    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:35:07.109033 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:07.108943    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:35:07.697250 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:07.693094    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:35:07.938098 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:07.938064    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:35:08.650123 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:08.650009    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:35:10.820349 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:10.820281    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:35:11.117151 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:11.117049    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-553753/distinct-2-w2-553753-kserve-5bf8568844-7q4l2" podUID="a1894b80-164c-4ce1-94f3-1e5011d9759e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Nov 03 22:35:11.167239 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:11.162773    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/premium-w0-88a4d9-kserve-5559d75b7b-vqllq" podUID="05206737-4ca3-41fc-b35a-37c625cc182c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:35:12.690316 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:12.690276    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:35:12.846242 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:12.844440    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:35:13.152737 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:13.152651    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:35:14.330756 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:14.330722    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:35:14.471953 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:14.471915    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:35:14.485905 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:14.485872    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-65b0e2-5fc849b4cf-m2md4"]
Nov 03 22:35:14.490622 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:14.490604    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-65b0e2-5fc849b4cf-m2md4"
Nov 03 22:35:14.507351 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:14.507314    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-65b0e2\""
Nov 03 22:35:14.510507 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:14.510484    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-65b0e2-5fc849b4cf-m2md4"]
Nov 03 22:35:14.517125 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:14.517101    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-65b0e2-dockercfg-cx678\""
Nov 03 22:35:14.642295 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:14.640988    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgxbs\" (UniqueName: \"kubernetes.io/projected/c907cfe8-b91c-4539-8503-20eb6561f3ad-kube-api-access-qgxbs\") pod \"payload-pre-processing-e2e-api-a-65b0e2-5fc849b4cf-m2md4\" (UID: \"c907cfe8-b91c-4539-8503-20eb6561f3ad\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-65b0e2-5fc849b4cf-m2md4"
Nov 03 22:35:14.642295 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:14.641325    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c907cfe8-b91c-4539-8503-20eb6561f3ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-65b0e2-5fc849b4cf-m2md4\" (UID: \"c907cfe8-b91c-4539-8503-20eb6561f3ad\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-65b0e2-5fc849b4cf-m2md4"
Nov 03 22:35:14.719303 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:14.719266    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-65b0e2-765dd6d498-5n54k"]
Nov 03 22:35:14.766318 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:14.753055    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgxbs\" (UniqueName: \"kubernetes.io/projected/c907cfe8-b91c-4539-8503-20eb6561f3ad-kube-api-access-qgxbs\") pod \"payload-pre-processing-e2e-api-a-65b0e2-5fc849b4cf-m2md4\" (UID: \"c907cfe8-b91c-4539-8503-20eb6561f3ad\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-65b0e2-5fc849b4cf-m2md4"
Nov 03 22:35:14.766318 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:14.753455    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c907cfe8-b91c-4539-8503-20eb6561f3ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-65b0e2-5fc849b4cf-m2md4\" (UID: \"c907cfe8-b91c-4539-8503-20eb6561f3ad\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-65b0e2-5fc849b4cf-m2md4"
Nov 03 22:35:14.766318 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:14.754526    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c907cfe8-b91c-4539-8503-20eb6561f3ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-65b0e2-5fc849b4cf-m2md4\" (UID: \"c907cfe8-b91c-4539-8503-20eb6561f3ad\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-65b0e2-5fc849b4cf-m2md4"
Nov 03 22:35:14.846237 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:14.820172    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-65b0e2-765dd6d498-5n54k"
Nov 03 22:35:14.903297 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:14.866755    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-65b0e2-765dd6d498-5n54k"]
Nov 03 22:35:14.981938 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:14.964996    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6mxx\" (UniqueName: \"kubernetes.io/projected/22f9e7cf-d329-4178-af61-b9c27d485907-kube-api-access-l6mxx\") pod \"payload-processing-e2e-api-a-65b0e2-765dd6d498-5n54k\" (UID: \"22f9e7cf-d329-4178-af61-b9c27d485907\") " pod="openshift-ingress/payload-processing-e2e-api-a-65b0e2-765dd6d498-5n54k"
Nov 03 22:35:14.985484 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:14.985438    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22f9e7cf-d329-4178-af61-b9c27d485907-plugins-config-volume\") pod \"payload-processing-e2e-api-a-65b0e2-765dd6d498-5n54k\" (UID: \"22f9e7cf-d329-4178-af61-b9c27d485907\") " pod="openshift-ingress/payload-processing-e2e-api-a-65b0e2-765dd6d498-5n54k"
Nov 03 22:35:14.986337 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:14.979107    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgxbs\" (UniqueName: \"kubernetes.io/projected/c907cfe8-b91c-4539-8503-20eb6561f3ad-kube-api-access-qgxbs\") pod \"payload-pre-processing-e2e-api-a-65b0e2-5fc849b4cf-m2md4\" (UID: \"c907cfe8-b91c-4539-8503-20eb6561f3ad\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-65b0e2-5fc849b4cf-m2md4"
Nov 03 22:35:15.087725 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:15.087693    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6mxx\" (UniqueName: \"kubernetes.io/projected/22f9e7cf-d329-4178-af61-b9c27d485907-kube-api-access-l6mxx\") pod \"payload-processing-e2e-api-a-65b0e2-765dd6d498-5n54k\" (UID: \"22f9e7cf-d329-4178-af61-b9c27d485907\") " pod="openshift-ingress/payload-processing-e2e-api-a-65b0e2-765dd6d498-5n54k"
Nov 03 22:35:15.087936 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:15.087777    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22f9e7cf-d329-4178-af61-b9c27d485907-plugins-config-volume\") pod \"payload-processing-e2e-api-a-65b0e2-765dd6d498-5n54k\" (UID: \"22f9e7cf-d329-4178-af61-b9c27d485907\") " pod="openshift-ingress/payload-processing-e2e-api-a-65b0e2-765dd6d498-5n54k"
Nov 03 22:35:15.088555 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:15.088531    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22f9e7cf-d329-4178-af61-b9c27d485907-plugins-config-volume\") pod \"payload-processing-e2e-api-a-65b0e2-765dd6d498-5n54k\" (UID: \"22f9e7cf-d329-4178-af61-b9c27d485907\") " pod="openshift-ingress/payload-processing-e2e-api-a-65b0e2-765dd6d498-5n54k"
Nov 03 22:35:15.105540 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:15.105509    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-65b0e2-5fc849b4cf-m2md4"
Nov 03 22:35:15.112818 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:15.112787    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6mxx\" (UniqueName: \"kubernetes.io/projected/22f9e7cf-d329-4178-af61-b9c27d485907-kube-api-access-l6mxx\") pod \"payload-processing-e2e-api-a-65b0e2-765dd6d498-5n54k\" (UID: \"22f9e7cf-d329-4178-af61-b9c27d485907\") " pod="openshift-ingress/payload-processing-e2e-api-a-65b0e2-765dd6d498-5n54k"
Nov 03 22:35:15.183668 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:15.183636    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v"]
Nov 03 22:35:15.183895 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:15.183870    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v" podUID="1e928b67-e047-414b-92a2-8fc0792eed46" containerName="istio-proxy" containerID="cri-o://96198a4dd8065a2d9031b23b150da2e49e6eff60785e6216f90929120506961a" gracePeriod=30
Nov 03 22:35:15.303417 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:15.303388    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-65b0e2-765dd6d498-5n54k"
Nov 03 22:35:15.334237 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:15.334115    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:35:15.500997 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:15.500908    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x"]
Nov 03 22:35:15.502654 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:15.502611    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x" podUID="73504251-0779-459e-abfc-3d099b7a1339" containerName="istio-proxy" containerID="cri-o://bc05b9fb5baab8bfd239be42d22c2cc43804bc1cee4aa1a735eecc33c7843551" gracePeriod=30
Nov 03 22:35:15.635094 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:15.635058    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g"]
Nov 03 22:35:15.635406 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:15.635377    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g" podUID="0cd86bdd-8465-4676-9491-2266a2e86b3a" containerName="istio-proxy" containerID="cri-o://7c805bc5f4e30bc755a2ea2fd94a1b6bbb3af20626acabc892dfd21ee7c6879a" gracePeriod=30
Nov 03 22:35:15.676572 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:15.675270    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2"]
Nov 03 22:35:15.676572 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:15.675527    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2" podUID="8ace43d0-fe92-4e5e-a190-26408d4ab47b" containerName="istio-proxy" containerID="cri-o://249324c1e5ce56378c577f6b78a20dc22cbe2aeffc0d5624b4e10a8e9b73399c" gracePeriod=30
Nov 03 22:35:15.684447 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:15.684362    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk"]
Nov 03 22:35:15.684836 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:15.684804    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk" podUID="b92034e4-f537-425b-b28b-1cd528b700af" containerName="istio-proxy" containerID="cri-o://c67ac168dfce0b0498f3b2d6508b0683ebf6181f618986efd1eebcdf46ddb89c" gracePeriod=30
Nov 03 22:35:15.695997 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:15.695792    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-65b0e2-5fc849b4cf-m2md4"]
Nov 03 22:35:15.771588 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:15.770867    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-65b0e2-765dd6d498-5n54k"]
Nov 03 22:35:15.775206 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:35:15.775173    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22f9e7cf_d329_4178_af61_b9c27d485907.slice/crio-38fa57259914628e5ac9794ae78edfe9fc6d18fbde4a5d3a17a0155e1cd06f6c WatchSource:0}: Error finding container 38fa57259914628e5ac9794ae78edfe9fc6d18fbde4a5d3a17a0155e1cd06f6c: Status 404 returned error can't find the container with id 38fa57259914628e5ac9794ae78edfe9fc6d18fbde4a5d3a17a0155e1cd06f6c
Nov 03 22:35:15.876239 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:15.876181    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf"]
Nov 03 22:35:15.876460 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:15.876423    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf" podUID="489884ba-7c24-48f7-8a7e-e5494d3232f7" containerName="istio-proxy" containerID="cri-o://ff3b35ba33964b4d4718feb61bf79020cd3b07bb1453cd2898a3dc84892f8327" gracePeriod=30
Nov 03 22:35:15.997624 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:15.995771    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-bbfc2e-6ddb5f6584-hc4kx"
Nov 03 22:35:16.504798 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:16.504699    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-65b0e2-765dd6d498-5n54k" event={"ID":"22f9e7cf-d329-4178-af61-b9c27d485907","Type":"ContainerStarted","Data":"9970259817a937954e15c1b8b5a919a3c6ff32ee87efdb304cbaf9d12347f7f3"}
Nov 03 22:35:16.504798 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:16.504755    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-65b0e2-765dd6d498-5n54k" event={"ID":"22f9e7cf-d329-4178-af61-b9c27d485907","Type":"ContainerStarted","Data":"38fa57259914628e5ac9794ae78edfe9fc6d18fbde4a5d3a17a0155e1cd06f6c"}
Nov 03 22:35:16.504798 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:16.504781    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-65b0e2-765dd6d498-5n54k"
Nov 03 22:35:16.506684 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:16.506656    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-65b0e2-5fc849b4cf-m2md4" event={"ID":"c907cfe8-b91c-4539-8503-20eb6561f3ad","Type":"ContainerStarted","Data":"44f147fc2e0e54f09b61b3a27bee7e6ba7f0fa224cb5a2e65b1a05fb31fa7cc2"}
Nov 03 22:35:16.506810 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:16.506694    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-65b0e2-5fc849b4cf-m2md4" event={"ID":"c907cfe8-b91c-4539-8503-20eb6561f3ad","Type":"ContainerStarted","Data":"92fb86cd2a5e1e32f51b43e26b58d03110a31ec0d3d1a84615f80c854eaf74f5"}
Nov 03 22:35:16.506810 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:16.506762    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-65b0e2-5fc849b4cf-m2md4"
Nov 03 22:35:16.528553 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:16.528526    4431 patch_prober.go:29] interesting pod/e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 22:35:16.528689 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:16.528581    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk" podUID="b92034e4-f537-425b-b28b-1cd528b700af" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 22:35:16.546002 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:16.545951    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-65b0e2-765dd6d498-5n54k" podStartSLOduration=2.545937256 podStartE2EDuration="2.545937256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:35:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:35:16.544136648 +0000 UTC m=+1340.569992511" watchObservedRunningTime="2026-11-03 22:35:16.545937256 +0000 UTC m=+1340.571793118"
Nov 03 22:35:16.590636 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:16.590577    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-65b0e2-5fc849b4cf-m2md4" podStartSLOduration=2.590560349 podStartE2EDuration="2.590560349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:35:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:35:16.587159591 +0000 UTC m=+1340.613015453" watchObservedRunningTime="2026-11-03 22:35:16.590560349 +0000 UTC m=+1340.616416212"
Nov 03 22:35:17.032930 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:17.032853    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:35:18.620589 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:18.620550    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:35:20.207445 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:20.207377    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:35:20.519314 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:20.519287    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v"
Nov 03 22:35:20.541271 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:20.541240    4431 generic.go:356] "Generic (PLEG): container finished" podID="f44ff83c-d054-4f20-a04c-c6204de2bae2" containerID="fd8992222dab0b30bb72530b1349c352047bd2118d3f30adca4cb7b736c3d0cb" exitCode=0
Nov 03 22:35:20.541399 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:20.541307    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb" event={"ID":"f44ff83c-d054-4f20-a04c-c6204de2bae2","Type":"ContainerDied","Data":"fd8992222dab0b30bb72530b1349c352047bd2118d3f30adca4cb7b736c3d0cb"}
Nov 03 22:35:20.541816 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:20.541796    4431 scope.go:122] "RemoveContainer" containerID="fd8992222dab0b30bb72530b1349c352047bd2118d3f30adca4cb7b736c3d0cb"
Nov 03 22:35:20.542901 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:20.542882    4431 generic.go:356] "Generic (PLEG): container finished" podID="1e928b67-e047-414b-92a2-8fc0792eed46" containerID="96198a4dd8065a2d9031b23b150da2e49e6eff60785e6216f90929120506961a" exitCode=0
Nov 03 22:35:20.542973 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:20.542931    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v" event={"ID":"1e928b67-e047-414b-92a2-8fc0792eed46","Type":"ContainerDied","Data":"96198a4dd8065a2d9031b23b150da2e49e6eff60785e6216f90929120506961a"}
Nov 03 22:35:20.542973 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:20.542934    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v"
Nov 03 22:35:20.542973 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:20.542961    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v" event={"ID":"1e928b67-e047-414b-92a2-8fc0792eed46","Type":"ContainerDied","Data":"717447bedadc16d9cfb7cb3b2ef3344bfcb11a7e1839e3e476ec07c6155c273e"}
Nov 03 22:35:20.543122 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:20.542981    4431 scope.go:122] "RemoveContainer" containerID="96198a4dd8065a2d9031b23b150da2e49e6eff60785e6216f90929120506961a"
Nov 03 22:35:20.543859 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:20.543831    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:35:20.543953 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:20.543910    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:35:20.544049 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:20.543955    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:35:20.630188 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:20.630162    4431 scope.go:122] "RemoveContainer" containerID="96198a4dd8065a2d9031b23b150da2e49e6eff60785e6216f90929120506961a"
Nov 03 22:35:20.630772 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:35:20.630722    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96198a4dd8065a2d9031b23b150da2e49e6eff60785e6216f90929120506961a\": container with ID starting with 96198a4dd8065a2d9031b23b150da2e49e6eff60785e6216f90929120506961a not found: ID does not exist" containerID="96198a4dd8065a2d9031b23b150da2e49e6eff60785e6216f90929120506961a"
Nov 03 22:35:20.630772 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:20.630759    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96198a4dd8065a2d9031b23b150da2e49e6eff60785e6216f90929120506961a"} err="failed to get container status \"96198a4dd8065a2d9031b23b150da2e49e6eff60785e6216f90929120506961a\": rpc error: code = NotFound desc = could not find container \"96198a4dd8065a2d9031b23b150da2e49e6eff60785e6216f90929120506961a\": container with ID starting with 96198a4dd8065a2d9031b23b150da2e49e6eff60785e6216f90929120506961a not found: ID does not exist"
Nov 03 22:35:20.653130 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:20.652355    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e928b67-e047-414b-92a2-8fc0792eed46-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e928b67-e047-414b-92a2-8fc0792eed46-workload-socket\") pod \"1e928b67-e047-414b-92a2-8fc0792eed46\" (UID: \"1e928b67-e047-414b-92a2-8fc0792eed46\") "
Nov 03 22:35:20.653130 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:20.652509    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e928b67-e047-414b-92a2-8fc0792eed46-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e928b67-e047-414b-92a2-8fc0792eed46-workload-certs\") pod \"1e928b67-e047-414b-92a2-8fc0792eed46\" (UID: \"1e928b67-e047-414b-92a2-8fc0792eed46\") "
Nov 03 22:35:20.653130 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:20.652585    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e928b67-e047-414b-92a2-8fc0792eed46-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e928b67-e047-414b-92a2-8fc0792eed46-credential-socket\") pod \"1e928b67-e047-414b-92a2-8fc0792eed46\" (UID: \"1e928b67-e047-414b-92a2-8fc0792eed46\") "
Nov 03 22:35:20.653130 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:20.652626    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e928b67-e047-414b-92a2-8fc0792eed46-kube-api-access-qlmlf\" (UniqueName: \"kubernetes.io/projected/1e928b67-e047-414b-92a2-8fc0792eed46-kube-api-access-qlmlf\") pod \"1e928b67-e047-414b-92a2-8fc0792eed46\" (UID: \"1e928b67-e047-414b-92a2-8fc0792eed46\") "
Nov 03 22:35:20.653130 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:20.652676    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1e928b67-e047-414b-92a2-8fc0792eed46-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e928b67-e047-414b-92a2-8fc0792eed46-istio-podinfo\") pod \"1e928b67-e047-414b-92a2-8fc0792eed46\" (UID: \"1e928b67-e047-414b-92a2-8fc0792eed46\") "
Nov 03 22:35:20.653130 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:20.652719    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e928b67-e047-414b-92a2-8fc0792eed46-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e928b67-e047-414b-92a2-8fc0792eed46-istiod-ca-cert\") pod \"1e928b67-e047-414b-92a2-8fc0792eed46\" (UID: \"1e928b67-e047-414b-92a2-8fc0792eed46\") "
Nov 03 22:35:20.653130 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:20.652858    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e928b67-e047-414b-92a2-8fc0792eed46-istio-token\" (UniqueName: \"kubernetes.io/projected/1e928b67-e047-414b-92a2-8fc0792eed46-istio-token\") pod \"1e928b67-e047-414b-92a2-8fc0792eed46\" (UID: \"1e928b67-e047-414b-92a2-8fc0792eed46\") "
Nov 03 22:35:20.653130 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:20.653037    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e928b67-e047-414b-92a2-8fc0792eed46-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e928b67-e047-414b-92a2-8fc0792eed46-istio-envoy\") pod \"1e928b67-e047-414b-92a2-8fc0792eed46\" (UID: \"1e928b67-e047-414b-92a2-8fc0792eed46\") "
Nov 03 22:35:20.653130 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:20.653091    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e928b67-e047-414b-92a2-8fc0792eed46-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e928b67-e047-414b-92a2-8fc0792eed46-istio-data\") pod \"1e928b67-e047-414b-92a2-8fc0792eed46\" (UID: \"1e928b67-e047-414b-92a2-8fc0792eed46\") "
Nov 03 22:35:20.655629 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:20.655601    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e928b67-e047-414b-92a2-8fc0792eed46-istio-data" pod "1e928b67-e047-414b-92a2-8fc0792eed46" (UID: "1e928b67-e047-414b-92a2-8fc0792eed46"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:35:20.656328 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:20.655917    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e928b67-e047-414b-92a2-8fc0792eed46-workload-socket" pod "1e928b67-e047-414b-92a2-8fc0792eed46" (UID: "1e928b67-e047-414b-92a2-8fc0792eed46"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:35:20.657080 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:20.656871    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e928b67-e047-414b-92a2-8fc0792eed46-istiod-ca-cert" pod "1e928b67-e047-414b-92a2-8fc0792eed46" (UID: "1e928b67-e047-414b-92a2-8fc0792eed46"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:35:20.657413 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:20.657247    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e928b67-e047-414b-92a2-8fc0792eed46-workload-certs" pod "1e928b67-e047-414b-92a2-8fc0792eed46" (UID: "1e928b67-e047-414b-92a2-8fc0792eed46"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:35:20.657561 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:20.657534    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e928b67-e047-414b-92a2-8fc0792eed46-credential-socket" pod "1e928b67-e047-414b-92a2-8fc0792eed46" (UID: "1e928b67-e047-414b-92a2-8fc0792eed46"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:35:20.659276 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:20.659257    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1e928b67-e047-414b-92a2-8fc0792eed46-istio-podinfo" pod "1e928b67-e047-414b-92a2-8fc0792eed46" (UID: "1e928b67-e047-414b-92a2-8fc0792eed46"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 22:35:20.664575 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:20.663969    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e928b67-e047-414b-92a2-8fc0792eed46-istio-token" pod "1e928b67-e047-414b-92a2-8fc0792eed46" (UID: "1e928b67-e047-414b-92a2-8fc0792eed46"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:35:20.665556 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:20.665535    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e928b67-e047-414b-92a2-8fc0792eed46-istio-envoy" pod "1e928b67-e047-414b-92a2-8fc0792eed46" (UID: "1e928b67-e047-414b-92a2-8fc0792eed46"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:35:20.665711 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:20.665567    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e928b67-e047-414b-92a2-8fc0792eed46-kube-api-access-qlmlf" pod "1e928b67-e047-414b-92a2-8fc0792eed46" (UID: "1e928b67-e047-414b-92a2-8fc0792eed46"). InnerVolumeSpecName "kube-api-access-qlmlf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:35:20.758253 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:20.756272    4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e928b67-e047-414b-92a2-8fc0792eed46-istio-envoy\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:35:20.758253 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:20.756306    4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e928b67-e047-414b-92a2-8fc0792eed46-istio-data\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:35:20.758253 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:20.756321    4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e928b67-e047-414b-92a2-8fc0792eed46-workload-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:35:20.758253 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:20.756334    4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e928b67-e047-414b-92a2-8fc0792eed46-workload-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:35:20.758253 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:20.756351    4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e928b67-e047-414b-92a2-8fc0792eed46-credential-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:35:20.758253 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:20.756365    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlmlf\" (UniqueName: \"kubernetes.io/projected/1e928b67-e047-414b-92a2-8fc0792eed46-kube-api-access-qlmlf\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:35:20.758253 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:20.756379    4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e928b67-e047-414b-92a2-8fc0792eed46-istio-podinfo\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:35:20.758253 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:20.756392    4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e928b67-e047-414b-92a2-8fc0792eed46-istiod-ca-cert\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:35:20.758253 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:20.756403    4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e928b67-e047-414b-92a2-8fc0792eed46-istio-token\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:35:20.820689 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:20.820660    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x"
Nov 03 22:35:20.988237 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:20.979206    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73504251-0779-459e-abfc-3d099b7a1339-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73504251-0779-459e-abfc-3d099b7a1339-istio-data\") pod \"73504251-0779-459e-abfc-3d099b7a1339\" (UID: \"73504251-0779-459e-abfc-3d099b7a1339\") "
Nov 03 22:35:20.988237 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:20.979287    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73504251-0779-459e-abfc-3d099b7a1339-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73504251-0779-459e-abfc-3d099b7a1339-istio-envoy\") pod \"73504251-0779-459e-abfc-3d099b7a1339\" (UID: \"73504251-0779-459e-abfc-3d099b7a1339\") "
Nov 03 22:35:20.988237 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:20.979375    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73504251-0779-459e-abfc-3d099b7a1339-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73504251-0779-459e-abfc-3d099b7a1339-workload-certs\") pod \"73504251-0779-459e-abfc-3d099b7a1339\" (UID: \"73504251-0779-459e-abfc-3d099b7a1339\") "
Nov 03 22:35:20.988237 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:20.979424    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73504251-0779-459e-abfc-3d099b7a1339-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73504251-0779-459e-abfc-3d099b7a1339-istiod-ca-cert\") pod \"73504251-0779-459e-abfc-3d099b7a1339\" (UID: \"73504251-0779-459e-abfc-3d099b7a1339\") "
Nov 03 22:35:20.988237 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:20.979457    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/73504251-0779-459e-abfc-3d099b7a1339-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73504251-0779-459e-abfc-3d099b7a1339-istio-podinfo\") pod \"73504251-0779-459e-abfc-3d099b7a1339\" (UID: \"73504251-0779-459e-abfc-3d099b7a1339\") "
Nov 03 22:35:20.988237 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:20.979545    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73504251-0779-459e-abfc-3d099b7a1339-kube-api-access-8p95m\" (UniqueName: \"kubernetes.io/projected/73504251-0779-459e-abfc-3d099b7a1339-kube-api-access-8p95m\") pod \"73504251-0779-459e-abfc-3d099b7a1339\" (UID: \"73504251-0779-459e-abfc-3d099b7a1339\") "
Nov 03 22:35:20.988237 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:20.979634    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73504251-0779-459e-abfc-3d099b7a1339-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73504251-0779-459e-abfc-3d099b7a1339-workload-socket\") pod \"73504251-0779-459e-abfc-3d099b7a1339\" (UID: \"73504251-0779-459e-abfc-3d099b7a1339\") "
Nov 03 22:35:20.988237 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:20.979661    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73504251-0779-459e-abfc-3d099b7a1339-istio-token\" (UniqueName: \"kubernetes.io/projected/73504251-0779-459e-abfc-3d099b7a1339-istio-token\") pod \"73504251-0779-459e-abfc-3d099b7a1339\" (UID: \"73504251-0779-459e-abfc-3d099b7a1339\") "
Nov 03 22:35:20.988237 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:20.979744    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73504251-0779-459e-abfc-3d099b7a1339-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73504251-0779-459e-abfc-3d099b7a1339-credential-socket\") pod \"73504251-0779-459e-abfc-3d099b7a1339\" (UID: \"73504251-0779-459e-abfc-3d099b7a1339\") "
Nov 03 22:35:20.988237 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:20.980365    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73504251-0779-459e-abfc-3d099b7a1339-istiod-ca-cert" pod "73504251-0779-459e-abfc-3d099b7a1339" (UID: "73504251-0779-459e-abfc-3d099b7a1339"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:35:20.988237 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:20.980584    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73504251-0779-459e-abfc-3d099b7a1339-credential-socket" pod "73504251-0779-459e-abfc-3d099b7a1339" (UID: "73504251-0779-459e-abfc-3d099b7a1339"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:35:20.988237 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:20.980640    4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73504251-0779-459e-abfc-3d099b7a1339-istiod-ca-cert\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:35:20.988237 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:20.980871    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73504251-0779-459e-abfc-3d099b7a1339-istio-data" pod "73504251-0779-459e-abfc-3d099b7a1339" (UID: "73504251-0779-459e-abfc-3d099b7a1339"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:35:20.988237 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:20.980979    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73504251-0779-459e-abfc-3d099b7a1339-workload-certs" pod "73504251-0779-459e-abfc-3d099b7a1339" (UID: "73504251-0779-459e-abfc-3d099b7a1339"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:35:20.988237 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:20.981099    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73504251-0779-459e-abfc-3d099b7a1339-workload-socket" pod "73504251-0779-459e-abfc-3d099b7a1339" (UID: "73504251-0779-459e-abfc-3d099b7a1339"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:35:20.988237 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:20.984519    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73504251-0779-459e-abfc-3d099b7a1339-istio-envoy" pod "73504251-0779-459e-abfc-3d099b7a1339" (UID: "73504251-0779-459e-abfc-3d099b7a1339"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:35:21.000627 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.000441    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g"
Nov 03 22:35:21.009896 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.009853    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v"]
Nov 03 22:35:21.011863 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.011820    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/73504251-0779-459e-abfc-3d099b7a1339-istio-podinfo" pod "73504251-0779-459e-abfc-3d099b7a1339" (UID: "73504251-0779-459e-abfc-3d099b7a1339"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 22:35:21.012303 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.012278    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73504251-0779-459e-abfc-3d099b7a1339-kube-api-access-8p95m" pod "73504251-0779-459e-abfc-3d099b7a1339" (UID: "73504251-0779-459e-abfc-3d099b7a1339"). InnerVolumeSpecName "kube-api-access-8p95m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:35:21.012564 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.012537    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73504251-0779-459e-abfc-3d099b7a1339-istio-token" pod "73504251-0779-459e-abfc-3d099b7a1339" (UID: "73504251-0779-459e-abfc-3d099b7a1339"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:35:21.019370 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.019341    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-2jb6v"]
Nov 03 22:35:21.086270 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.082859    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cd86bdd-8465-4676-9491-2266a2e86b3a-kube-api-access-pwjx4\" (UniqueName: \"kubernetes.io/projected/0cd86bdd-8465-4676-9491-2266a2e86b3a-kube-api-access-pwjx4\") pod \"0cd86bdd-8465-4676-9491-2266a2e86b3a\" (UID: \"0cd86bdd-8465-4676-9491-2266a2e86b3a\") "
Nov 03 22:35:21.086270 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.082947    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cd86bdd-8465-4676-9491-2266a2e86b3a-istio-token\" (UniqueName: \"kubernetes.io/projected/0cd86bdd-8465-4676-9491-2266a2e86b3a-istio-token\") pod \"0cd86bdd-8465-4676-9491-2266a2e86b3a\" (UID: \"0cd86bdd-8465-4676-9491-2266a2e86b3a\") "
Nov 03 22:35:21.086270 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.083039    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cd86bdd-8465-4676-9491-2266a2e86b3a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cd86bdd-8465-4676-9491-2266a2e86b3a-workload-socket\") pod \"0cd86bdd-8465-4676-9491-2266a2e86b3a\" (UID: \"0cd86bdd-8465-4676-9491-2266a2e86b3a\") "
Nov 03 22:35:21.086270 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.083141    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cd86bdd-8465-4676-9491-2266a2e86b3a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cd86bdd-8465-4676-9491-2266a2e86b3a-istiod-ca-cert\") pod \"0cd86bdd-8465-4676-9491-2266a2e86b3a\" (UID: \"0cd86bdd-8465-4676-9491-2266a2e86b3a\") "
Nov 03 22:35:21.086270 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.083299    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0cd86bdd-8465-4676-9491-2266a2e86b3a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cd86bdd-8465-4676-9491-2266a2e86b3a-istio-podinfo\") pod \"0cd86bdd-8465-4676-9491-2266a2e86b3a\" (UID: \"0cd86bdd-8465-4676-9491-2266a2e86b3a\") "
Nov 03 22:35:21.086270 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.083356    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cd86bdd-8465-4676-9491-2266a2e86b3a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cd86bdd-8465-4676-9491-2266a2e86b3a-credential-socket\") pod \"0cd86bdd-8465-4676-9491-2266a2e86b3a\" (UID: \"0cd86bdd-8465-4676-9491-2266a2e86b3a\") "
Nov 03 22:35:21.086270 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.083484    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cd86bdd-8465-4676-9491-2266a2e86b3a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cd86bdd-8465-4676-9491-2266a2e86b3a-workload-certs\") pod \"0cd86bdd-8465-4676-9491-2266a2e86b3a\" (UID: \"0cd86bdd-8465-4676-9491-2266a2e86b3a\") "
Nov 03 22:35:21.086270 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.083653    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cd86bdd-8465-4676-9491-2266a2e86b3a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cd86bdd-8465-4676-9491-2266a2e86b3a-istio-data\") pod \"0cd86bdd-8465-4676-9491-2266a2e86b3a\" (UID: \"0cd86bdd-8465-4676-9491-2266a2e86b3a\") "
Nov 03 22:35:21.086270 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.083762    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cd86bdd-8465-4676-9491-2266a2e86b3a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cd86bdd-8465-4676-9491-2266a2e86b3a-istio-envoy\") pod \"0cd86bdd-8465-4676-9491-2266a2e86b3a\" (UID: \"0cd86bdd-8465-4676-9491-2266a2e86b3a\") "
Nov 03 22:35:21.086270 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.084667    4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73504251-0779-459e-abfc-3d099b7a1339-istio-data\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:35:21.086270 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.084689    4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73504251-0779-459e-abfc-3d099b7a1339-istio-envoy\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:35:21.086270 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.084703    4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73504251-0779-459e-abfc-3d099b7a1339-workload-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:35:21.086270 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.084715    4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73504251-0779-459e-abfc-3d099b7a1339-istio-podinfo\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:35:21.086270 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.084729    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8p95m\" (UniqueName: \"kubernetes.io/projected/73504251-0779-459e-abfc-3d099b7a1339-kube-api-access-8p95m\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:35:21.086270 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.084745    4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73504251-0779-459e-abfc-3d099b7a1339-workload-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:35:21.086270 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.084758    4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73504251-0779-459e-abfc-3d099b7a1339-istio-token\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:35:21.086270 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.084773    4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73504251-0779-459e-abfc-3d099b7a1339-credential-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:35:21.086270 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.085805    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cd86bdd-8465-4676-9491-2266a2e86b3a-credential-socket" pod "0cd86bdd-8465-4676-9491-2266a2e86b3a" (UID: "0cd86bdd-8465-4676-9491-2266a2e86b3a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:35:21.086270 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.085905    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cd86bdd-8465-4676-9491-2266a2e86b3a-istiod-ca-cert" pod "0cd86bdd-8465-4676-9491-2266a2e86b3a" (UID: "0cd86bdd-8465-4676-9491-2266a2e86b3a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:35:21.087705 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.086432    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cd86bdd-8465-4676-9491-2266a2e86b3a-workload-socket" pod "0cd86bdd-8465-4676-9491-2266a2e86b3a" (UID: "0cd86bdd-8465-4676-9491-2266a2e86b3a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:35:21.090877 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.090786    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cd86bdd-8465-4676-9491-2266a2e86b3a-istio-data" pod "0cd86bdd-8465-4676-9491-2266a2e86b3a" (UID: "0cd86bdd-8465-4676-9491-2266a2e86b3a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:35:21.091908 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.091454    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cd86bdd-8465-4676-9491-2266a2e86b3a-workload-certs" pod "0cd86bdd-8465-4676-9491-2266a2e86b3a" (UID: "0cd86bdd-8465-4676-9491-2266a2e86b3a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:35:21.092578 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.092484    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cd86bdd-8465-4676-9491-2266a2e86b3a-istio-token" pod "0cd86bdd-8465-4676-9491-2266a2e86b3a" (UID: "0cd86bdd-8465-4676-9491-2266a2e86b3a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:35:21.092578 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.092546    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cd86bdd-8465-4676-9491-2266a2e86b3a-istio-envoy" pod "0cd86bdd-8465-4676-9491-2266a2e86b3a" (UID: "0cd86bdd-8465-4676-9491-2266a2e86b3a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:35:21.110300 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.110064    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0cd86bdd-8465-4676-9491-2266a2e86b3a-istio-podinfo" pod "0cd86bdd-8465-4676-9491-2266a2e86b3a" (UID: "0cd86bdd-8465-4676-9491-2266a2e86b3a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 22:35:21.110300 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.110193    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cd86bdd-8465-4676-9491-2266a2e86b3a-kube-api-access-pwjx4" pod "0cd86bdd-8465-4676-9491-2266a2e86b3a" (UID: "0cd86bdd-8465-4676-9491-2266a2e86b3a"). InnerVolumeSpecName "kube-api-access-pwjx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:35:21.160198 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.159819    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2"
Nov 03 22:35:21.205133 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.203064    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwjx4\" (UniqueName: \"kubernetes.io/projected/0cd86bdd-8465-4676-9491-2266a2e86b3a-kube-api-access-pwjx4\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:35:21.205133 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.203102    4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cd86bdd-8465-4676-9491-2266a2e86b3a-istio-token\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:35:21.205133 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.203118    4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cd86bdd-8465-4676-9491-2266a2e86b3a-workload-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:35:21.205133 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.203130    4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cd86bdd-8465-4676-9491-2266a2e86b3a-istiod-ca-cert\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:35:21.205133 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.203143    4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cd86bdd-8465-4676-9491-2266a2e86b3a-istio-podinfo\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:35:21.205133 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.203156    4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cd86bdd-8465-4676-9491-2266a2e86b3a-credential-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:35:21.205133 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.203168    4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cd86bdd-8465-4676-9491-2266a2e86b3a-workload-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:35:21.205133 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.203180    4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cd86bdd-8465-4676-9491-2266a2e86b3a-istio-data\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:35:21.205133 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.203194    4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cd86bdd-8465-4676-9491-2266a2e86b3a-istio-envoy\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:35:21.239268 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.239046    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk"
Nov 03 22:35:21.304631 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.304603    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ace43d0-fe92-4e5e-a190-26408d4ab47b-istio-token\" (UniqueName: \"kubernetes.io/projected/8ace43d0-fe92-4e5e-a190-26408d4ab47b-istio-token\") pod \"8ace43d0-fe92-4e5e-a190-26408d4ab47b\" (UID: \"8ace43d0-fe92-4e5e-a190-26408d4ab47b\") "
Nov 03 22:35:21.305079 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.305058    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ace43d0-fe92-4e5e-a190-26408d4ab47b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ace43d0-fe92-4e5e-a190-26408d4ab47b-credential-socket\") pod \"8ace43d0-fe92-4e5e-a190-26408d4ab47b\" (UID: \"8ace43d0-fe92-4e5e-a190-26408d4ab47b\") "
Nov 03 22:35:21.305256 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.305176    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ace43d0-fe92-4e5e-a190-26408d4ab47b-kube-api-access-67fzs\" (UniqueName: \"kubernetes.io/projected/8ace43d0-fe92-4e5e-a190-26408d4ab47b-kube-api-access-67fzs\") pod \"8ace43d0-fe92-4e5e-a190-26408d4ab47b\" (UID: \"8ace43d0-fe92-4e5e-a190-26408d4ab47b\") "
Nov 03 22:35:21.305320 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.305268    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ace43d0-fe92-4e5e-a190-26408d4ab47b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ace43d0-fe92-4e5e-a190-26408d4ab47b-istio-data\") pod \"8ace43d0-fe92-4e5e-a190-26408d4ab47b\" (UID: \"8ace43d0-fe92-4e5e-a190-26408d4ab47b\") "
Nov 03 22:35:21.305379 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.305319    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ace43d0-fe92-4e5e-a190-26408d4ab47b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ace43d0-fe92-4e5e-a190-26408d4ab47b-istiod-ca-cert\") pod \"8ace43d0-fe92-4e5e-a190-26408d4ab47b\" (UID: \"8ace43d0-fe92-4e5e-a190-26408d4ab47b\") "
Nov 03 22:35:21.305427 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.305377    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ace43d0-fe92-4e5e-a190-26408d4ab47b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ace43d0-fe92-4e5e-a190-26408d4ab47b-workload-socket\") pod \"8ace43d0-fe92-4e5e-a190-26408d4ab47b\" (UID: \"8ace43d0-fe92-4e5e-a190-26408d4ab47b\") "
Nov 03 22:35:21.305497 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.305481    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8ace43d0-fe92-4e5e-a190-26408d4ab47b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ace43d0-fe92-4e5e-a190-26408d4ab47b-istio-podinfo\") pod \"8ace43d0-fe92-4e5e-a190-26408d4ab47b\" (UID: \"8ace43d0-fe92-4e5e-a190-26408d4ab47b\") "
Nov 03 22:35:21.305561 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.305524    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ace43d0-fe92-4e5e-a190-26408d4ab47b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ace43d0-fe92-4e5e-a190-26408d4ab47b-workload-certs\") pod \"8ace43d0-fe92-4e5e-a190-26408d4ab47b\" (UID: \"8ace43d0-fe92-4e5e-a190-26408d4ab47b\") "
Nov 03 22:35:21.305637 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.305623    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ace43d0-fe92-4e5e-a190-26408d4ab47b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ace43d0-fe92-4e5e-a190-26408d4ab47b-istio-envoy\") pod \"8ace43d0-fe92-4e5e-a190-26408d4ab47b\" (UID: \"8ace43d0-fe92-4e5e-a190-26408d4ab47b\") "
Nov 03 22:35:21.305898 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.305878    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ace43d0-fe92-4e5e-a190-26408d4ab47b-istio-data" pod "8ace43d0-fe92-4e5e-a190-26408d4ab47b" (UID: "8ace43d0-fe92-4e5e-a190-26408d4ab47b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:35:21.307561 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.306575    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ace43d0-fe92-4e5e-a190-26408d4ab47b-credential-socket" pod "8ace43d0-fe92-4e5e-a190-26408d4ab47b" (UID: "8ace43d0-fe92-4e5e-a190-26408d4ab47b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:35:21.307561 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.306868    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ace43d0-fe92-4e5e-a190-26408d4ab47b-workload-certs" pod "8ace43d0-fe92-4e5e-a190-26408d4ab47b" (UID: "8ace43d0-fe92-4e5e-a190-26408d4ab47b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:35:21.308459 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.308074    4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ace43d0-fe92-4e5e-a190-26408d4ab47b-istio-data\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:35:21.308566 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.308459    4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ace43d0-fe92-4e5e-a190-26408d4ab47b-credential-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:35:21.308566 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.307009    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ace43d0-fe92-4e5e-a190-26408d4ab47b-workload-socket" pod "8ace43d0-fe92-4e5e-a190-26408d4ab47b" (UID: "8ace43d0-fe92-4e5e-a190-26408d4ab47b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:35:21.309674 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.308852    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ace43d0-fe92-4e5e-a190-26408d4ab47b-istiod-ca-cert" pod "8ace43d0-fe92-4e5e-a190-26408d4ab47b" (UID: "8ace43d0-fe92-4e5e-a190-26408d4ab47b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:35:21.310854 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.310816    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ace43d0-fe92-4e5e-a190-26408d4ab47b-istio-envoy" pod "8ace43d0-fe92-4e5e-a190-26408d4ab47b" (UID: "8ace43d0-fe92-4e5e-a190-26408d4ab47b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:35:21.312354 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.312315    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8ace43d0-fe92-4e5e-a190-26408d4ab47b-istio-podinfo" pod "8ace43d0-fe92-4e5e-a190-26408d4ab47b" (UID: "8ace43d0-fe92-4e5e-a190-26408d4ab47b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 22:35:21.312625 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.312599    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ace43d0-fe92-4e5e-a190-26408d4ab47b-istio-token" pod "8ace43d0-fe92-4e5e-a190-26408d4ab47b" (UID: "8ace43d0-fe92-4e5e-a190-26408d4ab47b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:35:21.314539 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.314170    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf"
Nov 03 22:35:21.314539 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.314311    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ace43d0-fe92-4e5e-a190-26408d4ab47b-kube-api-access-67fzs" pod "8ace43d0-fe92-4e5e-a190-26408d4ab47b" (UID: "8ace43d0-fe92-4e5e-a190-26408d4ab47b"). InnerVolumeSpecName "kube-api-access-67fzs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:35:21.409233 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.409190    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b92034e4-f537-425b-b28b-1cd528b700af-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b92034e4-f537-425b-b28b-1cd528b700af-workload-certs\") pod \"b92034e4-f537-425b-b28b-1cd528b700af\" (UID: \"b92034e4-f537-425b-b28b-1cd528b700af\") "
Nov 03 22:35:21.409430 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.409314    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/489884ba-7c24-48f7-8a7e-e5494d3232f7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/489884ba-7c24-48f7-8a7e-e5494d3232f7-istiod-ca-cert\") pod \"489884ba-7c24-48f7-8a7e-e5494d3232f7\" (UID: \"489884ba-7c24-48f7-8a7e-e5494d3232f7\") "
Nov 03 22:35:21.409430 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.409336    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/489884ba-7c24-48f7-8a7e-e5494d3232f7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/489884ba-7c24-48f7-8a7e-e5494d3232f7-istio-envoy\") pod \"489884ba-7c24-48f7-8a7e-e5494d3232f7\" (UID: \"489884ba-7c24-48f7-8a7e-e5494d3232f7\") "
Nov 03 22:35:21.409430 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.409398    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/489884ba-7c24-48f7-8a7e-e5494d3232f7-istio-token\" (UniqueName: \"kubernetes.io/projected/489884ba-7c24-48f7-8a7e-e5494d3232f7-istio-token\") pod \"489884ba-7c24-48f7-8a7e-e5494d3232f7\" (UID: \"489884ba-7c24-48f7-8a7e-e5494d3232f7\") "
Nov 03 22:35:21.409610 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.409486    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b92034e4-f537-425b-b28b-1cd528b700af-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b92034e4-f537-425b-b28b-1cd528b700af-credential-socket\") pod \"b92034e4-f537-425b-b28b-1cd528b700af\" (UID: \"b92034e4-f537-425b-b28b-1cd528b700af\") "
Nov 03 22:35:21.409610 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.409552    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b92034e4-f537-425b-b28b-1cd528b700af-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b92034e4-f537-425b-b28b-1cd528b700af-workload-socket\") pod \"b92034e4-f537-425b-b28b-1cd528b700af\" (UID: \"b92034e4-f537-425b-b28b-1cd528b700af\") "
Nov 03 22:35:21.409789 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.409622    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b92034e4-f537-425b-b28b-1cd528b700af-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b92034e4-f537-425b-b28b-1cd528b700af-istio-data\") pod \"b92034e4-f537-425b-b28b-1cd528b700af\" (UID: \"b92034e4-f537-425b-b28b-1cd528b700af\") "
Nov 03 22:35:21.409789 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.409692    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/489884ba-7c24-48f7-8a7e-e5494d3232f7-istiod-ca-cert" pod "489884ba-7c24-48f7-8a7e-e5494d3232f7" (UID: "489884ba-7c24-48f7-8a7e-e5494d3232f7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:35:21.409789 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.409734    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b92034e4-f537-425b-b28b-1cd528b700af-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b92034e4-f537-425b-b28b-1cd528b700af-istio-podinfo\") pod \"b92034e4-f537-425b-b28b-1cd528b700af\" (UID: \"b92034e4-f537-425b-b28b-1cd528b700af\") "
Nov 03 22:35:21.409789 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.409777    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/489884ba-7c24-48f7-8a7e-e5494d3232f7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/489884ba-7c24-48f7-8a7e-e5494d3232f7-credential-socket\") pod \"489884ba-7c24-48f7-8a7e-e5494d3232f7\" (UID: \"489884ba-7c24-48f7-8a7e-e5494d3232f7\") "
Nov 03 22:35:21.410016 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.409830    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/489884ba-7c24-48f7-8a7e-e5494d3232f7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/489884ba-7c24-48f7-8a7e-e5494d3232f7-istio-podinfo\") pod \"489884ba-7c24-48f7-8a7e-e5494d3232f7\" (UID: \"489884ba-7c24-48f7-8a7e-e5494d3232f7\") "
Nov 03 22:35:21.410016 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.409892    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/489884ba-7c24-48f7-8a7e-e5494d3232f7-kube-api-access-6c5wt\" (UniqueName: \"kubernetes.io/projected/489884ba-7c24-48f7-8a7e-e5494d3232f7-kube-api-access-6c5wt\") pod \"489884ba-7c24-48f7-8a7e-e5494d3232f7\" (UID: \"489884ba-7c24-48f7-8a7e-e5494d3232f7\") "
Nov 03 22:35:21.410016 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.409937    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b92034e4-f537-425b-b28b-1cd528b700af-istio-token\" (UniqueName: \"kubernetes.io/projected/b92034e4-f537-425b-b28b-1cd528b700af-istio-token\") pod \"b92034e4-f537-425b-b28b-1cd528b700af\" (UID: \"b92034e4-f537-425b-b28b-1cd528b700af\") "
Nov 03 22:35:21.410016 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.409981    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b92034e4-f537-425b-b28b-1cd528b700af-credential-socket" pod "b92034e4-f537-425b-b28b-1cd528b700af" (UID: "b92034e4-f537-425b-b28b-1cd528b700af"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:35:21.410016 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.409991    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b92034e4-f537-425b-b28b-1cd528b700af-kube-api-access-t4gcw\" (UniqueName: \"kubernetes.io/projected/b92034e4-f537-425b-b28b-1cd528b700af-kube-api-access-t4gcw\") pod \"b92034e4-f537-425b-b28b-1cd528b700af\" (UID: \"b92034e4-f537-425b-b28b-1cd528b700af\") "
Nov 03 22:35:21.410352 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.410008    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b92034e4-f537-425b-b28b-1cd528b700af-istio-data" pod "b92034e4-f537-425b-b28b-1cd528b700af" (UID: "b92034e4-f537-425b-b28b-1cd528b700af"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:35:21.410352 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.410027    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/489884ba-7c24-48f7-8a7e-e5494d3232f7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/489884ba-7c24-48f7-8a7e-e5494d3232f7-istio-data\") pod \"489884ba-7c24-48f7-8a7e-e5494d3232f7\" (UID: \"489884ba-7c24-48f7-8a7e-e5494d3232f7\") "
Nov 03 22:35:21.410352 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.410118    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b92034e4-f537-425b-b28b-1cd528b700af-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b92034e4-f537-425b-b28b-1cd528b700af-istio-envoy\") pod \"b92034e4-f537-425b-b28b-1cd528b700af\" (UID: \"b92034e4-f537-425b-b28b-1cd528b700af\") "
Nov 03 22:35:21.410352 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.410160    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b92034e4-f537-425b-b28b-1cd528b700af-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b92034e4-f537-425b-b28b-1cd528b700af-istiod-ca-cert\") pod \"b92034e4-f537-425b-b28b-1cd528b700af\" (UID: \"b92034e4-f537-425b-b28b-1cd528b700af\") "
Nov 03 22:35:21.410352 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.410169    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b92034e4-f537-425b-b28b-1cd528b700af-workload-socket" pod "b92034e4-f537-425b-b28b-1cd528b700af" (UID: "b92034e4-f537-425b-b28b-1cd528b700af"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:35:21.410352 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.410188    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/489884ba-7c24-48f7-8a7e-e5494d3232f7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/489884ba-7c24-48f7-8a7e-e5494d3232f7-workload-socket\") pod \"489884ba-7c24-48f7-8a7e-e5494d3232f7\" (UID: \"489884ba-7c24-48f7-8a7e-e5494d3232f7\") "
Nov 03 22:35:21.410352 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.410244    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/489884ba-7c24-48f7-8a7e-e5494d3232f7-istio-data" pod "489884ba-7c24-48f7-8a7e-e5494d3232f7" (UID: "489884ba-7c24-48f7-8a7e-e5494d3232f7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:35:21.410352 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.410260    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/489884ba-7c24-48f7-8a7e-e5494d3232f7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/489884ba-7c24-48f7-8a7e-e5494d3232f7-workload-certs\") pod \"489884ba-7c24-48f7-8a7e-e5494d3232f7\" (UID: \"489884ba-7c24-48f7-8a7e-e5494d3232f7\") "
Nov 03 22:35:21.410764 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.410384    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/489884ba-7c24-48f7-8a7e-e5494d3232f7-workload-socket" pod "489884ba-7c24-48f7-8a7e-e5494d3232f7" (UID: "489884ba-7c24-48f7-8a7e-e5494d3232f7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:35:21.410764 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.410402    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/489884ba-7c24-48f7-8a7e-e5494d3232f7-credential-socket" pod "489884ba-7c24-48f7-8a7e-e5494d3232f7" (UID: "489884ba-7c24-48f7-8a7e-e5494d3232f7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:35:21.410878 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.410738    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b92034e4-f537-425b-b28b-1cd528b700af-workload-certs" pod "b92034e4-f537-425b-b28b-1cd528b700af" (UID: "b92034e4-f537-425b-b28b-1cd528b700af"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:35:21.411792 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.410983    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/489884ba-7c24-48f7-8a7e-e5494d3232f7-workload-certs" pod "489884ba-7c24-48f7-8a7e-e5494d3232f7" (UID: "489884ba-7c24-48f7-8a7e-e5494d3232f7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:35:21.411792 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.410776    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b92034e4-f537-425b-b28b-1cd528b700af-istiod-ca-cert" pod "b92034e4-f537-425b-b28b-1cd528b700af" (UID: "b92034e4-f537-425b-b28b-1cd528b700af"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:35:21.411792 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.411310    4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ace43d0-fe92-4e5e-a190-26408d4ab47b-istiod-ca-cert\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:35:21.411792 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.411334    4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ace43d0-fe92-4e5e-a190-26408d4ab47b-workload-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:35:21.411792 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.411349    4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b92034e4-f537-425b-b28b-1cd528b700af-credential-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:35:21.411792 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.411367    4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b92034e4-f537-425b-b28b-1cd528b700af-workload-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:35:21.411792 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.411381    4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ace43d0-fe92-4e5e-a190-26408d4ab47b-istio-podinfo\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:35:21.411792 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.411395    4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ace43d0-fe92-4e5e-a190-26408d4ab47b-workload-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:35:21.411792 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.411407    4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b92034e4-f537-425b-b28b-1cd528b700af-istio-data\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:35:21.411792 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.411418    4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ace43d0-fe92-4e5e-a190-26408d4ab47b-istio-envoy\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:35:21.411792 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.411430    4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/489884ba-7c24-48f7-8a7e-e5494d3232f7-credential-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:35:21.411792 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.411443    4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ace43d0-fe92-4e5e-a190-26408d4ab47b-istio-token\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:35:21.411792 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.411455    4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/489884ba-7c24-48f7-8a7e-e5494d3232f7-istio-data\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:35:21.411792 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.411468    4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b92034e4-f537-425b-b28b-1cd528b700af-istiod-ca-cert\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:35:21.411792 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.411481    4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/489884ba-7c24-48f7-8a7e-e5494d3232f7-workload-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:35:21.411792 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.411494    4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/489884ba-7c24-48f7-8a7e-e5494d3232f7-workload-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:35:21.411792 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.411507    4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b92034e4-f537-425b-b28b-1cd528b700af-workload-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:35:21.411792 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.411521    4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/489884ba-7c24-48f7-8a7e-e5494d3232f7-istiod-ca-cert\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:35:21.411792 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.411534    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67fzs\" (UniqueName: \"kubernetes.io/projected/8ace43d0-fe92-4e5e-a190-26408d4ab47b-kube-api-access-67fzs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:35:21.413610 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.413573    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b92034e4-f537-425b-b28b-1cd528b700af-istio-podinfo" pod "b92034e4-f537-425b-b28b-1cd528b700af" (UID: "b92034e4-f537-425b-b28b-1cd528b700af"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 22:35:21.414047 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.413850    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/489884ba-7c24-48f7-8a7e-e5494d3232f7-istio-token" pod "489884ba-7c24-48f7-8a7e-e5494d3232f7" (UID: "489884ba-7c24-48f7-8a7e-e5494d3232f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:35:21.414047 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.413916    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/489884ba-7c24-48f7-8a7e-e5494d3232f7-istio-podinfo" pod "489884ba-7c24-48f7-8a7e-e5494d3232f7" (UID: "489884ba-7c24-48f7-8a7e-e5494d3232f7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 22:35:21.414047 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.414019    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/489884ba-7c24-48f7-8a7e-e5494d3232f7-istio-envoy" pod "489884ba-7c24-48f7-8a7e-e5494d3232f7" (UID: "489884ba-7c24-48f7-8a7e-e5494d3232f7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:35:21.414735 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.414710    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b92034e4-f537-425b-b28b-1cd528b700af-istio-token" pod "b92034e4-f537-425b-b28b-1cd528b700af" (UID: "b92034e4-f537-425b-b28b-1cd528b700af"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:35:21.414988 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.414964    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b92034e4-f537-425b-b28b-1cd528b700af-istio-envoy" pod "b92034e4-f537-425b-b28b-1cd528b700af" (UID: "b92034e4-f537-425b-b28b-1cd528b700af"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:35:21.414988 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.414978    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/489884ba-7c24-48f7-8a7e-e5494d3232f7-kube-api-access-6c5wt" pod "489884ba-7c24-48f7-8a7e-e5494d3232f7" (UID: "489884ba-7c24-48f7-8a7e-e5494d3232f7"). InnerVolumeSpecName "kube-api-access-6c5wt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:35:21.415313 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.415292    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b92034e4-f537-425b-b28b-1cd528b700af-kube-api-access-t4gcw" pod "b92034e4-f537-425b-b28b-1cd528b700af" (UID: "b92034e4-f537-425b-b28b-1cd528b700af"). InnerVolumeSpecName "kube-api-access-t4gcw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:35:21.512846 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.512797    4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b92034e4-f537-425b-b28b-1cd528b700af-istio-podinfo\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:35:21.513009 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.512841    4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/489884ba-7c24-48f7-8a7e-e5494d3232f7-istio-podinfo\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:35:21.513009 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.512874    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6c5wt\" (UniqueName: \"kubernetes.io/projected/489884ba-7c24-48f7-8a7e-e5494d3232f7-kube-api-access-6c5wt\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:35:21.513009 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.512884    4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b92034e4-f537-425b-b28b-1cd528b700af-istio-token\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:35:21.513009 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.512893    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4gcw\" (UniqueName: \"kubernetes.io/projected/b92034e4-f537-425b-b28b-1cd528b700af-kube-api-access-t4gcw\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:35:21.513009 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.512901    4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b92034e4-f537-425b-b28b-1cd528b700af-istio-envoy\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:35:21.513009 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.512909    4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/489884ba-7c24-48f7-8a7e-e5494d3232f7-istio-envoy\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:35:21.513009 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.512917    4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/489884ba-7c24-48f7-8a7e-e5494d3232f7-istio-token\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:35:21.562938 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.562894    4431 generic.go:356] "Generic (PLEG): container finished" podID="0cd86bdd-8465-4676-9491-2266a2e86b3a" containerID="7c805bc5f4e30bc755a2ea2fd94a1b6bbb3af20626acabc892dfd21ee7c6879a" exitCode=0
Nov 03 22:35:21.563124 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.562963    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g"
Nov 03 22:35:21.563124 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.562975    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g" event={"ID":"0cd86bdd-8465-4676-9491-2266a2e86b3a","Type":"ContainerDied","Data":"7c805bc5f4e30bc755a2ea2fd94a1b6bbb3af20626acabc892dfd21ee7c6879a"}
Nov 03 22:35:21.563124 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.563067    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g" event={"ID":"0cd86bdd-8465-4676-9491-2266a2e86b3a","Type":"ContainerDied","Data":"765eb6e4cf6372c519ef96b493bf320d8fda3da591cebbebe27873b8b59697b6"}
Nov 03 22:35:21.563124 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.563094    4431 scope.go:122] "RemoveContainer" containerID="7c805bc5f4e30bc755a2ea2fd94a1b6bbb3af20626acabc892dfd21ee7c6879a"
Nov 03 22:35:21.564548 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.564512    4431 generic.go:356] "Generic (PLEG): container finished" podID="8ace43d0-fe92-4e5e-a190-26408d4ab47b" containerID="249324c1e5ce56378c577f6b78a20dc22cbe2aeffc0d5624b4e10a8e9b73399c" exitCode=0
Nov 03 22:35:21.564621 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.564543    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2" event={"ID":"8ace43d0-fe92-4e5e-a190-26408d4ab47b","Type":"ContainerDied","Data":"249324c1e5ce56378c577f6b78a20dc22cbe2aeffc0d5624b4e10a8e9b73399c"}
Nov 03 22:35:21.564621 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.564575    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2"
Nov 03 22:35:21.564621 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.564595    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2" event={"ID":"8ace43d0-fe92-4e5e-a190-26408d4ab47b","Type":"ContainerDied","Data":"0fb918dd53bfd27fa404b9570b969fa8212a467e7a59d47fb0eb33bff7b283c3"}
Nov 03 22:35:21.566082 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.566059    4431 generic.go:356] "Generic (PLEG): container finished" podID="73504251-0779-459e-abfc-3d099b7a1339" containerID="bc05b9fb5baab8bfd239be42d22c2cc43804bc1cee4aa1a735eecc33c7843551" exitCode=0
Nov 03 22:35:21.566255 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.566115    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x"
Nov 03 22:35:21.566255 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.566141    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x" event={"ID":"73504251-0779-459e-abfc-3d099b7a1339","Type":"ContainerDied","Data":"bc05b9fb5baab8bfd239be42d22c2cc43804bc1cee4aa1a735eecc33c7843551"}
Nov 03 22:35:21.566255 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.566187    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x" event={"ID":"73504251-0779-459e-abfc-3d099b7a1339","Type":"ContainerDied","Data":"dca6fff42a51abfd2c1a824cc935a29a5b191827346c33c560090b6364fce937"}
Nov 03 22:35:21.588465 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.587456    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb" event={"ID":"f44ff83c-d054-4f20-a04c-c6204de2bae2","Type":"ContainerStarted","Data":"42a5da8b666c75ce24ff12a58e032fdc19ad1897c395692e3852b6f7cb45284e"}
Nov 03 22:35:21.592082 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.592051    4431 generic.go:356] "Generic (PLEG): container finished" podID="489884ba-7c24-48f7-8a7e-e5494d3232f7" containerID="ff3b35ba33964b4d4718feb61bf79020cd3b07bb1453cd2898a3dc84892f8327" exitCode=0
Nov 03 22:35:21.592248 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.592085    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf" event={"ID":"489884ba-7c24-48f7-8a7e-e5494d3232f7","Type":"ContainerDied","Data":"ff3b35ba33964b4d4718feb61bf79020cd3b07bb1453cd2898a3dc84892f8327"}
Nov 03 22:35:21.592248 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.592118    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf"
Nov 03 22:35:21.592248 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.592135    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf" event={"ID":"489884ba-7c24-48f7-8a7e-e5494d3232f7","Type":"ContainerDied","Data":"86b4dc28811306b6ef13e8039612661250b580a7fd0b1c8a3c6dd03ee1eeab6b"}
Nov 03 22:35:21.592248 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.592170    4431 scope.go:122] "RemoveContainer" containerID="7c805bc5f4e30bc755a2ea2fd94a1b6bbb3af20626acabc892dfd21ee7c6879a"
Nov 03 22:35:21.592907 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:35:21.592876    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c805bc5f4e30bc755a2ea2fd94a1b6bbb3af20626acabc892dfd21ee7c6879a\": container with ID starting with 7c805bc5f4e30bc755a2ea2fd94a1b6bbb3af20626acabc892dfd21ee7c6879a not found: ID does not exist" containerID="7c805bc5f4e30bc755a2ea2fd94a1b6bbb3af20626acabc892dfd21ee7c6879a"
Nov 03 22:35:21.593121 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.592914    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c805bc5f4e30bc755a2ea2fd94a1b6bbb3af20626acabc892dfd21ee7c6879a"} err="failed to get container status \"7c805bc5f4e30bc755a2ea2fd94a1b6bbb3af20626acabc892dfd21ee7c6879a\": rpc error: code = NotFound desc = could not find container \"7c805bc5f4e30bc755a2ea2fd94a1b6bbb3af20626acabc892dfd21ee7c6879a\": container with ID starting with 7c805bc5f4e30bc755a2ea2fd94a1b6bbb3af20626acabc892dfd21ee7c6879a not found: ID does not exist"
Nov 03 22:35:21.593121 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.592952    4431 scope.go:122] "RemoveContainer" containerID="249324c1e5ce56378c577f6b78a20dc22cbe2aeffc0d5624b4e10a8e9b73399c"
Nov 03 22:35:21.595310 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.595287    4431 generic.go:356] "Generic (PLEG): container finished" podID="b92034e4-f537-425b-b28b-1cd528b700af" containerID="c67ac168dfce0b0498f3b2d6508b0683ebf6181f618986efd1eebcdf46ddb89c" exitCode=0
Nov 03 22:35:21.595392 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.595344    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk"
Nov 03 22:35:21.595392 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.595353    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk" event={"ID":"b92034e4-f537-425b-b28b-1cd528b700af","Type":"ContainerDied","Data":"c67ac168dfce0b0498f3b2d6508b0683ebf6181f618986efd1eebcdf46ddb89c"}
Nov 03 22:35:21.595392 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.595389    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk" event={"ID":"b92034e4-f537-425b-b28b-1cd528b700af","Type":"ContainerDied","Data":"42cbe28309d690fa5bec84534d4b3aa5b282ea11ec4d4cdd462b01988af14aab"}
Nov 03 22:35:21.606890 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.606867    4431 scope.go:122] "RemoveContainer" containerID="249324c1e5ce56378c577f6b78a20dc22cbe2aeffc0d5624b4e10a8e9b73399c"
Nov 03 22:35:21.607159 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:35:21.607140    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"249324c1e5ce56378c577f6b78a20dc22cbe2aeffc0d5624b4e10a8e9b73399c\": container with ID starting with 249324c1e5ce56378c577f6b78a20dc22cbe2aeffc0d5624b4e10a8e9b73399c not found: ID does not exist" containerID="249324c1e5ce56378c577f6b78a20dc22cbe2aeffc0d5624b4e10a8e9b73399c"
Nov 03 22:35:21.607243 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.607172    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"249324c1e5ce56378c577f6b78a20dc22cbe2aeffc0d5624b4e10a8e9b73399c"} err="failed to get container status \"249324c1e5ce56378c577f6b78a20dc22cbe2aeffc0d5624b4e10a8e9b73399c\": rpc error: code = NotFound desc = could not find container \"249324c1e5ce56378c577f6b78a20dc22cbe2aeffc0d5624b4e10a8e9b73399c\": container with ID starting with 249324c1e5ce56378c577f6b78a20dc22cbe2aeffc0d5624b4e10a8e9b73399c not found: ID does not exist"
Nov 03 22:35:21.607243 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.607195    4431 scope.go:122] "RemoveContainer" containerID="bc05b9fb5baab8bfd239be42d22c2cc43804bc1cee4aa1a735eecc33c7843551"
Nov 03 22:35:21.618453 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.618437    4431 scope.go:122] "RemoveContainer" containerID="bc05b9fb5baab8bfd239be42d22c2cc43804bc1cee4aa1a735eecc33c7843551"
Nov 03 22:35:21.618718 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:35:21.618702    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc05b9fb5baab8bfd239be42d22c2cc43804bc1cee4aa1a735eecc33c7843551\": container with ID starting with bc05b9fb5baab8bfd239be42d22c2cc43804bc1cee4aa1a735eecc33c7843551 not found: ID does not exist" containerID="bc05b9fb5baab8bfd239be42d22c2cc43804bc1cee4aa1a735eecc33c7843551"
Nov 03 22:35:21.618765 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.618723    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc05b9fb5baab8bfd239be42d22c2cc43804bc1cee4aa1a735eecc33c7843551"} err="failed to get container status \"bc05b9fb5baab8bfd239be42d22c2cc43804bc1cee4aa1a735eecc33c7843551\": rpc error: code = NotFound desc = could not find container \"bc05b9fb5baab8bfd239be42d22c2cc43804bc1cee4aa1a735eecc33c7843551\": container with ID starting with bc05b9fb5baab8bfd239be42d22c2cc43804bc1cee4aa1a735eecc33c7843551 not found: ID does not exist"
Nov 03 22:35:21.618765 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.618738    4431 scope.go:122] "RemoveContainer" containerID="ff3b35ba33964b4d4718feb61bf79020cd3b07bb1453cd2898a3dc84892f8327"
Nov 03 22:35:21.628580 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.628559    4431 scope.go:122] "RemoveContainer" containerID="ff3b35ba33964b4d4718feb61bf79020cd3b07bb1453cd2898a3dc84892f8327"
Nov 03 22:35:21.628859 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:35:21.628832    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff3b35ba33964b4d4718feb61bf79020cd3b07bb1453cd2898a3dc84892f8327\": container with ID starting with ff3b35ba33964b4d4718feb61bf79020cd3b07bb1453cd2898a3dc84892f8327 not found: ID does not exist" containerID="ff3b35ba33964b4d4718feb61bf79020cd3b07bb1453cd2898a3dc84892f8327"
Nov 03 22:35:21.628938 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.628866    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff3b35ba33964b4d4718feb61bf79020cd3b07bb1453cd2898a3dc84892f8327"} err="failed to get container status \"ff3b35ba33964b4d4718feb61bf79020cd3b07bb1453cd2898a3dc84892f8327\": rpc error: code = NotFound desc = could not find container \"ff3b35ba33964b4d4718feb61bf79020cd3b07bb1453cd2898a3dc84892f8327\": container with ID starting with ff3b35ba33964b4d4718feb61bf79020cd3b07bb1453cd2898a3dc84892f8327 not found: ID does not exist"
Nov 03 22:35:21.628938 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.628882    4431 scope.go:122] "RemoveContainer" containerID="c67ac168dfce0b0498f3b2d6508b0683ebf6181f618986efd1eebcdf46ddb89c"
Nov 03 22:35:21.638394 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.638368    4431 scope.go:122] "RemoveContainer" containerID="c67ac168dfce0b0498f3b2d6508b0683ebf6181f618986efd1eebcdf46ddb89c"
Nov 03 22:35:21.638661 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:35:21.638641    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c67ac168dfce0b0498f3b2d6508b0683ebf6181f618986efd1eebcdf46ddb89c\": container with ID starting with c67ac168dfce0b0498f3b2d6508b0683ebf6181f618986efd1eebcdf46ddb89c not found: ID does not exist" containerID="c67ac168dfce0b0498f3b2d6508b0683ebf6181f618986efd1eebcdf46ddb89c"
Nov 03 22:35:21.638726 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.638667    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c67ac168dfce0b0498f3b2d6508b0683ebf6181f618986efd1eebcdf46ddb89c"} err="failed to get container status \"c67ac168dfce0b0498f3b2d6508b0683ebf6181f618986efd1eebcdf46ddb89c\": rpc error: code = NotFound desc = could not find container \"c67ac168dfce0b0498f3b2d6508b0683ebf6181f618986efd1eebcdf46ddb89c\": container with ID starting with c67ac168dfce0b0498f3b2d6508b0683ebf6181f618986efd1eebcdf46ddb89c not found: ID does not exist"
Nov 03 22:35:21.653576 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.653552    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g"]
Nov 03 22:35:21.659412 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.659390    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-4wl8g"]
Nov 03 22:35:21.692155 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.692103    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x"]
Nov 03 22:35:21.697204 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.697178    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cd47x"]
Nov 03 22:35:21.732607 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.732573    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf"]
Nov 03 22:35:21.748550 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.748505    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-x96rf"]
Nov 03 22:35:21.781207 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.781175    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk"]
Nov 03 22:35:21.785391 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.785359    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-6t8qk"]
Nov 03 22:35:21.817810 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.817777    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2"]
Nov 03 22:35:21.824145 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.824117    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-mqzl2"]
Nov 03 22:35:21.880832 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.880800    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb"
Nov 03 22:35:21.881951 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.881926    4431 patch_prober.go:29] interesting pod/e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Nov 03 22:35:21.882040 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.881982    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb" podUID="f44ff83c-d054-4f20-a04c-c6204de2bae2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Nov 03 22:35:21.968937 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:21.968852    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:35:22.095303 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:22.095267    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cd86bdd-8465-4676-9491-2266a2e86b3a" path="/var/lib/kubelet/pods/0cd86bdd-8465-4676-9491-2266a2e86b3a/volumes"
Nov 03 22:35:22.096049 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:22.096021    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e928b67-e047-414b-92a2-8fc0792eed46" path="/var/lib/kubelet/pods/1e928b67-e047-414b-92a2-8fc0792eed46/volumes"
Nov 03 22:35:22.096573 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:22.096556    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="489884ba-7c24-48f7-8a7e-e5494d3232f7" path="/var/lib/kubelet/pods/489884ba-7c24-48f7-8a7e-e5494d3232f7/volumes"
Nov 03 22:35:22.097618 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:22.097596    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73504251-0779-459e-abfc-3d099b7a1339" path="/var/lib/kubelet/pods/73504251-0779-459e-abfc-3d099b7a1339/volumes"
Nov 03 22:35:22.098123 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:22.098108    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ace43d0-fe92-4e5e-a190-26408d4ab47b" path="/var/lib/kubelet/pods/8ace43d0-fe92-4e5e-a190-26408d4ab47b/volumes"
Nov 03 22:35:22.098581 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:22.098568    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b92034e4-f537-425b-b28b-1cd528b700af" path="/var/lib/kubelet/pods/b92034e4-f537-425b-b28b-1cd528b700af/volumes"
Nov 03 22:35:22.225251 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:22.225150    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb"
Nov 03 22:35:22.881298 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:22.881208    4431 patch_prober.go:29] interesting pod/e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Nov 03 22:35:22.881708 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:22.881313    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb" podUID="f44ff83c-d054-4f20-a04c-c6204de2bae2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Nov 03 22:35:23.721235 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:23.716358    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:35:23.880323 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:23.880290    4431 patch_prober.go:29] interesting pod/e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Nov 03 22:35:23.880552 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:23.880366    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb" podUID="f44ff83c-d054-4f20-a04c-c6204de2bae2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Nov 03 22:35:24.880504 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:24.880470    4431 patch_prober.go:29] interesting pod/e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Nov 03 22:35:24.880929 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:24.880538    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb" podUID="f44ff83c-d054-4f20-a04c-c6204de2bae2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Nov 03 22:35:25.366961 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:25.366918    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:35:25.880886 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:25.880848    4431 patch_prober.go:29] interesting pod/e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Nov 03 22:35:25.881438 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:25.880929    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb" podUID="f44ff83c-d054-4f20-a04c-c6204de2bae2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Nov 03 22:35:26.880781 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:26.880744    4431 patch_prober.go:29] interesting pod/e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Nov 03 22:35:26.880976 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:26.880798    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb" podUID="f44ff83c-d054-4f20-a04c-c6204de2bae2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Nov 03 22:35:26.970339 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:26.970302    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-32fdf2-67d48ff746-m7rcv"]
Nov 03 22:35:26.971318 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:26.971292    4431 cpu_manager.go:414] "containerMap: removing container" podUID="b92034e4-f537-425b-b28b-1cd528b700af" containerName="istio-proxy"
Nov 03 22:35:26.971318 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:26.971317    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b92034e4-f537-425b-b28b-1cd528b700af" containerName="istio-proxy"
Nov 03 22:35:26.971501 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:26.971338    4431 cpu_manager.go:414] "containerMap: removing container" podUID="0cd86bdd-8465-4676-9491-2266a2e86b3a" containerName="istio-proxy"
Nov 03 22:35:26.971501 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:26.971346    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cd86bdd-8465-4676-9491-2266a2e86b3a" containerName="istio-proxy"
Nov 03 22:35:26.971501 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:26.971365    4431 cpu_manager.go:414] "containerMap: removing container" podUID="1e928b67-e047-414b-92a2-8fc0792eed46" containerName="istio-proxy"
Nov 03 22:35:26.971501 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:26.971371    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e928b67-e047-414b-92a2-8fc0792eed46" containerName="istio-proxy"
Nov 03 22:35:26.971501 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:26.971386    4431 cpu_manager.go:414] "containerMap: removing container" podUID="8ace43d0-fe92-4e5e-a190-26408d4ab47b" containerName="istio-proxy"
Nov 03 22:35:26.971501 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:26.971393    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ace43d0-fe92-4e5e-a190-26408d4ab47b" containerName="istio-proxy"
Nov 03 22:35:26.971501 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:26.971404    4431 cpu_manager.go:414] "containerMap: removing container" podUID="489884ba-7c24-48f7-8a7e-e5494d3232f7" containerName="istio-proxy"
Nov 03 22:35:26.971501 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:26.971412    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="489884ba-7c24-48f7-8a7e-e5494d3232f7" containerName="istio-proxy"
Nov 03 22:35:26.971501 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:26.971441    4431 cpu_manager.go:414] "containerMap: removing container" podUID="73504251-0779-459e-abfc-3d099b7a1339" containerName="istio-proxy"
Nov 03 22:35:26.971501 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:26.971446    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73504251-0779-459e-abfc-3d099b7a1339" containerName="istio-proxy"
Nov 03 22:35:26.971975 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:26.971594    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e928b67-e047-414b-92a2-8fc0792eed46" containerName="istio-proxy"
Nov 03 22:35:26.971975 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:26.971603    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ace43d0-fe92-4e5e-a190-26408d4ab47b" containerName="istio-proxy"
Nov 03 22:35:26.971975 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:26.971609    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="489884ba-7c24-48f7-8a7e-e5494d3232f7" containerName="istio-proxy"
Nov 03 22:35:26.971975 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:26.971618    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="73504251-0779-459e-abfc-3d099b7a1339" containerName="istio-proxy"
Nov 03 22:35:26.971975 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:26.971629    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="b92034e4-f537-425b-b28b-1cd528b700af" containerName="istio-proxy"
Nov 03 22:35:26.971975 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:26.971647    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cd86bdd-8465-4676-9491-2266a2e86b3a" containerName="istio-proxy"
Nov 03 22:35:26.976024 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:26.975998    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-32fdf2-67d48ff746-m7rcv"]
Nov 03 22:35:26.976162 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:26.976123    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-32fdf2-67d48ff746-m7rcv"
Nov 03 22:35:26.991342 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:26.991310    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-32fdf2-serving-cert\""
Nov 03 22:35:27.020546 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:27.020510    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55062db9-46d7-4c21-8240-ee7853f114f8-maas-api-tls\") pod \"maas-api-e2e-api-a-32fdf2-67d48ff746-m7rcv\" (UID: \"55062db9-46d7-4c21-8240-ee7853f114f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-32fdf2-67d48ff746-m7rcv"
Nov 03 22:35:27.020874 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:27.020617    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55062db9-46d7-4c21-8240-ee7853f114f8-openshift-service-ca\") pod \"maas-api-e2e-api-a-32fdf2-67d48ff746-m7rcv\" (UID: \"55062db9-46d7-4c21-8240-ee7853f114f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-32fdf2-67d48ff746-m7rcv"
Nov 03 22:35:27.020874 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:27.020752    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9g8m\" (UniqueName: \"kubernetes.io/projected/55062db9-46d7-4c21-8240-ee7853f114f8-kube-api-access-v9g8m\") pod \"maas-api-e2e-api-a-32fdf2-67d48ff746-m7rcv\" (UID: \"55062db9-46d7-4c21-8240-ee7853f114f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-32fdf2-67d48ff746-m7rcv"
Nov 03 22:35:27.020983 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:27.020937    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55062db9-46d7-4c21-8240-ee7853f114f8-metrics-certs\") pod \"maas-api-e2e-api-a-32fdf2-67d48ff746-m7rcv\" (UID: \"55062db9-46d7-4c21-8240-ee7853f114f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-32fdf2-67d48ff746-m7rcv"
Nov 03 22:35:27.094827 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:27.094793    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:35:27.122640 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:27.122608    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55062db9-46d7-4c21-8240-ee7853f114f8-openshift-service-ca\") pod \"maas-api-e2e-api-a-32fdf2-67d48ff746-m7rcv\" (UID: \"55062db9-46d7-4c21-8240-ee7853f114f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-32fdf2-67d48ff746-m7rcv"
Nov 03 22:35:27.122640 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:27.122645    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9g8m\" (UniqueName: \"kubernetes.io/projected/55062db9-46d7-4c21-8240-ee7853f114f8-kube-api-access-v9g8m\") pod \"maas-api-e2e-api-a-32fdf2-67d48ff746-m7rcv\" (UID: \"55062db9-46d7-4c21-8240-ee7853f114f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-32fdf2-67d48ff746-m7rcv"
Nov 03 22:35:27.122861 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:27.122761    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55062db9-46d7-4c21-8240-ee7853f114f8-metrics-certs\") pod \"maas-api-e2e-api-a-32fdf2-67d48ff746-m7rcv\" (UID: \"55062db9-46d7-4c21-8240-ee7853f114f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-32fdf2-67d48ff746-m7rcv"
Nov 03 22:35:27.123046 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:27.123025    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55062db9-46d7-4c21-8240-ee7853f114f8-maas-api-tls\") pod \"maas-api-e2e-api-a-32fdf2-67d48ff746-m7rcv\" (UID: \"55062db9-46d7-4c21-8240-ee7853f114f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-32fdf2-67d48ff746-m7rcv"
Nov 03 22:35:27.123367 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:27.123345    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55062db9-46d7-4c21-8240-ee7853f114f8-openshift-service-ca\") pod \"maas-api-e2e-api-a-32fdf2-67d48ff746-m7rcv\" (UID: \"55062db9-46d7-4c21-8240-ee7853f114f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-32fdf2-67d48ff746-m7rcv"
Nov 03 22:35:27.125373 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:27.125352    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55062db9-46d7-4c21-8240-ee7853f114f8-maas-api-tls\") pod \"maas-api-e2e-api-a-32fdf2-67d48ff746-m7rcv\" (UID: \"55062db9-46d7-4c21-8240-ee7853f114f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-32fdf2-67d48ff746-m7rcv"
Nov 03 22:35:27.125466 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:27.125439    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55062db9-46d7-4c21-8240-ee7853f114f8-metrics-certs\") pod \"maas-api-e2e-api-a-32fdf2-67d48ff746-m7rcv\" (UID: \"55062db9-46d7-4c21-8240-ee7853f114f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-32fdf2-67d48ff746-m7rcv"
Nov 03 22:35:27.159272 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:27.159151    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9g8m\" (UniqueName: \"kubernetes.io/projected/55062db9-46d7-4c21-8240-ee7853f114f8-kube-api-access-v9g8m\") pod \"maas-api-e2e-api-a-32fdf2-67d48ff746-m7rcv\" (UID: \"55062db9-46d7-4c21-8240-ee7853f114f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-32fdf2-67d48ff746-m7rcv"
Nov 03 22:35:27.247208 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:27.247157    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:35:27.305591 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:27.305561    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-32fdf2-67d48ff746-m7rcv"
Nov 03 22:35:27.520260 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:27.520200    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-65b0e2-5fc849b4cf-m2md4"
Nov 03 22:35:27.520434 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:27.520426    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-65b0e2-765dd6d498-5n54k"
Nov 03 22:35:27.628795 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:35:27.628746    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55062db9_46d7_4c21_8240_ee7853f114f8.slice/crio-e16c627b1733e69b761ed2cc707c7a6bf46d9e19cc7fc4ffa9a5e90615306d67 WatchSource:0}: Error finding container e16c627b1733e69b761ed2cc707c7a6bf46d9e19cc7fc4ffa9a5e90615306d67: Status 404 returned error can't find the container with id e16c627b1733e69b761ed2cc707c7a6bf46d9e19cc7fc4ffa9a5e90615306d67
Nov 03 22:35:27.631259 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:27.631234    4431 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 22:35:27.632329 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:27.632305    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-32fdf2-67d48ff746-m7rcv"]
Nov 03 22:35:27.651034 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:27.650991    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-32fdf2-67d48ff746-m7rcv" event={"ID":"55062db9-46d7-4c21-8240-ee7853f114f8","Type":"ContainerStarted","Data":"e16c627b1733e69b761ed2cc707c7a6bf46d9e19cc7fc4ffa9a5e90615306d67"}
Nov 03 22:35:27.885962 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:27.885866    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb"
Nov 03 22:35:27.887036 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:27.887014    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb"
Nov 03 22:35:28.660051 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:28.660007    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-32fdf2-67d48ff746-m7rcv" event={"ID":"55062db9-46d7-4c21-8240-ee7853f114f8","Type":"ContainerStarted","Data":"4b18608700ea09bd0496204ab100d6d9091255b92df92b52747deaf01caf6b3f"}
Nov 03 22:35:28.660330 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:28.660314    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-32fdf2-67d48ff746-m7rcv"
Nov 03 22:35:28.701876 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:28.701813    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-32fdf2-67d48ff746-m7rcv" podStartSLOduration=2.314095287 podStartE2EDuration="2.701794671s" totalImagesPullingTime="387.699384ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:35:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:35:28.700768065 +0000 UTC m=+1352.726623926" watchObservedRunningTime="2026-11-03 22:35:28.701794671 +0000 UTC m=+1352.727650534"
Nov 03 22:35:28.760857 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:28.760827    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:35:30.515766 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:30.515735    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:35:31.261299 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:35:31.261262    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:35:32.306856 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:32.306808    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:35:32.808431 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:32.808396    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:35:33.373236 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:33.372605    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss"]
Nov 03 22:35:33.378926 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:33.377771    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss"
Nov 03 22:35:33.404611 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:33.404580    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-4b5e2b-openshift-default-dockercfg-d2gcd\""
Nov 03 22:35:33.411234 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:33.407106    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss"]
Nov 03 22:35:33.477862 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:33.477689    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:35:33.493921 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:33.493515    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10b331d4-81e4-4243-a645-69d95958db00-istio-podinfo\") pod \"e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss\" (UID: \"10b331d4-81e4-4243-a645-69d95958db00\") " pod="openshift-ingress/e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss"
Nov 03 22:35:33.493921 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:33.493606    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10b331d4-81e4-4243-a645-69d95958db00-workload-certs\") pod \"e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss\" (UID: \"10b331d4-81e4-4243-a645-69d95958db00\") " pod="openshift-ingress/e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss"
Nov 03 22:35:33.493921 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:33.493711    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10b331d4-81e4-4243-a645-69d95958db00-istiod-ca-cert\") pod \"e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss\" (UID: \"10b331d4-81e4-4243-a645-69d95958db00\") " pod="openshift-ingress/e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss"
Nov 03 22:35:33.493921 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:33.493746    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnsrm\" (UniqueName: \"kubernetes.io/projected/10b331d4-81e4-4243-a645-69d95958db00-kube-api-access-lnsrm\") pod \"e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss\" (UID: \"10b331d4-81e4-4243-a645-69d95958db00\") " pod="openshift-ingress/e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss"
Nov 03 22:35:33.493921 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:33.493784    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10b331d4-81e4-4243-a645-69d95958db00-istio-data\") pod \"e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss\" (UID: \"10b331d4-81e4-4243-a645-69d95958db00\") " pod="openshift-ingress/e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss"
Nov 03 22:35:33.495151 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:33.494686    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10b331d4-81e4-4243-a645-69d95958db00-istio-envoy\") pod \"e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss\" (UID: \"10b331d4-81e4-4243-a645-69d95958db00\") " pod="openshift-ingress/e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss"
Nov 03 22:35:33.495151 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:33.494849    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b331d4-81e4-4243-a645-69d95958db00-workload-socket\") pod \"e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss\" (UID: \"10b331d4-81e4-4243-a645-69d95958db00\") " pod="openshift-ingress/e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss"
Nov 03 22:35:33.495151 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:33.494882    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b331d4-81e4-4243-a645-69d95958db00-credential-socket\") pod \"e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss\" (UID: \"10b331d4-81e4-4243-a645-69d95958db00\") " pod="openshift-ingress/e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss"
Nov 03 22:35:33.495151 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:33.494909    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10b331d4-81e4-4243-a645-69d95958db00-istio-token\") pod \"e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss\" (UID: \"10b331d4-81e4-4243-a645-69d95958db00\") " pod="openshift-ingress/e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss"
Nov 03 22:35:33.597099 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:33.597058    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10b331d4-81e4-4243-a645-69d95958db00-istio-envoy\") pod \"e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss\" (UID: \"10b331d4-81e4-4243-a645-69d95958db00\") " pod="openshift-ingress/e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss"
Nov 03 22:35:33.597301 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:33.597261    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b331d4-81e4-4243-a645-69d95958db00-workload-socket\") pod \"e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss\" (UID: \"10b331d4-81e4-4243-a645-69d95958db00\") " pod="openshift-ingress/e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss"
Nov 03 22:35:33.597351 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:33.597309    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b331d4-81e4-4243-a645-69d95958db00-credential-socket\") pod \"e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss\" (UID: \"10b331d4-81e4-4243-a645-69d95958db00\") " pod="openshift-ingress/e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss"
Nov 03 22:35:33.597351 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:33.597338    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10b331d4-81e4-4243-a645-69d95958db00-istio-token\") pod \"e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss\" (UID: \"10b331d4-81e4-4243-a645-69d95958db00\") " pod="openshift-ingress/e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss"
Nov 03 22:35:33.597558 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:33.597396    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10b331d4-81e4-4243-a645-69d95958db00-istio-podinfo\") pod \"e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss\" (UID: \"10b331d4-81e4-4243-a645-69d95958db00\") " pod="openshift-ingress/e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss"
Nov 03 22:35:33.597558 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:33.597485    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10b331d4-81e4-4243-a645-69d95958db00-workload-certs\") pod \"e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss\" (UID: \"10b331d4-81e4-4243-a645-69d95958db00\") " pod="openshift-ingress/e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss"
Nov 03 22:35:33.597664 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:33.597613    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10b331d4-81e4-4243-a645-69d95958db00-istiod-ca-cert\") pod \"e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss\" (UID: \"10b331d4-81e4-4243-a645-69d95958db00\") " pod="openshift-ingress/e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss"
Nov 03 22:35:33.597664 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:33.597652    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnsrm\" (UniqueName: \"kubernetes.io/projected/10b331d4-81e4-4243-a645-69d95958db00-kube-api-access-lnsrm\") pod \"e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss\" (UID: \"10b331d4-81e4-4243-a645-69d95958db00\") " pod="openshift-ingress/e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss"
Nov 03 22:35:33.597757 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:33.597690    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10b331d4-81e4-4243-a645-69d95958db00-istio-data\") pod \"e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss\" (UID: \"10b331d4-81e4-4243-a645-69d95958db00\") " pod="openshift-ingress/e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss"
Nov 03 22:35:33.597810 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:33.597748    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b331d4-81e4-4243-a645-69d95958db00-credential-socket\") pod \"e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss\" (UID: \"10b331d4-81e4-4243-a645-69d95958db00\") " pod="openshift-ingress/e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss"
Nov 03 22:35:33.597860 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:33.597818    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b331d4-81e4-4243-a645-69d95958db00-workload-socket\") pod \"e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss\" (UID: \"10b331d4-81e4-4243-a645-69d95958db00\") " pod="openshift-ingress/e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss"
Nov 03 22:35:33.598044 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:33.598019    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10b331d4-81e4-4243-a645-69d95958db00-istio-data\") pod \"e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss\" (UID: \"10b331d4-81e4-4243-a645-69d95958db00\") " pod="openshift-ingress/e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss"
Nov 03 22:35:33.598487 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:33.598464    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10b331d4-81e4-4243-a645-69d95958db00-workload-certs\") pod \"e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss\" (UID: \"10b331d4-81e4-4243-a645-69d95958db00\") " pod="openshift-ingress/e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss"
Nov 03 22:35:33.598744 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:33.598724    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10b331d4-81e4-4243-a645-69d95958db00-istiod-ca-cert\") pod \"e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss\" (UID: \"10b331d4-81e4-4243-a645-69d95958db00\") " pod="openshift-ingress/e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss"
Nov 03 22:35:33.600510 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:33.600485    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10b331d4-81e4-4243-a645-69d95958db00-istio-podinfo\") pod \"e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss\" (UID: \"10b331d4-81e4-4243-a645-69d95958db00\") " pod="openshift-ingress/e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss"
Nov 03 22:35:33.600836 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:33.600820    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10b331d4-81e4-4243-a645-69d95958db00-istio-envoy\") pod \"e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss\" (UID: \"10b331d4-81e4-4243-a645-69d95958db00\") " pod="openshift-ingress/e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss"
Nov 03 22:35:33.623706 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:33.623641    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10b331d4-81e4-4243-a645-69d95958db00-istio-token\") pod \"e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss\" (UID: \"10b331d4-81e4-4243-a645-69d95958db00\") " pod="openshift-ingress/e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss"
Nov 03 22:35:33.623831 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:33.623738    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnsrm\" (UniqueName: \"kubernetes.io/projected/10b331d4-81e4-4243-a645-69d95958db00-kube-api-access-lnsrm\") pod \"e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss\" (UID: \"10b331d4-81e4-4243-a645-69d95958db00\") " pod="openshift-ingress/e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss"
Nov 03 22:35:33.709089 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:33.709059    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss"
Nov 03 22:35:33.971416 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:33.971380    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:35:34.046773 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:34.046737    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss"]
Nov 03 22:35:34.049352 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:35:34.049324    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10b331d4_81e4_4243_a645_69d95958db00.slice/crio-9a0852a4a0ebd7cc9fb66d15521a3c16abd863fd23e74dd1f873e32001fe2925 WatchSource:0}: Error finding container 9a0852a4a0ebd7cc9fb66d15521a3c16abd863fd23e74dd1f873e32001fe2925: Status 404 returned error can't find the container with id 9a0852a4a0ebd7cc9fb66d15521a3c16abd863fd23e74dd1f873e32001fe2925
Nov 03 22:35:34.053433 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:34.053393    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:35:34.053511 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:34.053488    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:35:34.053600 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:34.053531    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:35:34.291582 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:34.291488    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:35:34.434331 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:34.434291    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:35:34.594958 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:34.594864    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:35:34.682636 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:34.682609    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-32fdf2-67d48ff746-m7rcv"
Nov 03 22:35:34.713635 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:34.713601    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss" event={"ID":"10b331d4-81e4-4243-a645-69d95958db00","Type":"ContainerStarted","Data":"9f24b059f48447fac205ac83277f336c7be557b4493c31260420aa82e575e556"}
Nov 03 22:35:34.713781 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:34.713663    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss" event={"ID":"10b331d4-81e4-4243-a645-69d95958db00","Type":"ContainerStarted","Data":"9a0852a4a0ebd7cc9fb66d15521a3c16abd863fd23e74dd1f873e32001fe2925"}
Nov 03 22:35:34.780652 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:34.780576    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss" podStartSLOduration=1.780557323 podStartE2EDuration="1.780557323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:35:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:35:34.772316883 +0000 UTC m=+1358.798172745" watchObservedRunningTime="2026-11-03 22:35:34.780557323 +0000 UTC m=+1358.806413187"
Nov 03 22:35:35.710185 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:35.710144    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss"
Nov 03 22:35:35.719578 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:35.719464    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss"
Nov 03 22:35:35.727781 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:35.727716    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss"
Nov 03 22:35:35.729099 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:35.729074    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss"
Nov 03 22:35:35.948836 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:35.948798    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:35:37.262436 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:37.262397    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:35:37.410430 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:37.410396    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:35:37.915289 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:37.899531    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:35:38.977808 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:38.977766    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:35:39.152437 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:39.152399    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:35:40.000511 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:40.000475    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:35:40.254034 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:40.253961    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:35:40.627859 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:40.627770    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8b1efeb6-5d7dd8ff94-q446n"]
Nov 03 22:35:40.684879 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:40.684840    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:35:40.685339 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:40.685320    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8b1efeb6-68787c86c9-wpjjk"]
Nov 03 22:35:40.881157 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:40.881026    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:35:41.073049 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:41.072952    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:35:41.251619 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:41.251578    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:35:42.130139 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:42.130103    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:35:43.764352 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:43.764318    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:35:45.528715 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:45.528683    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:35:45.629080 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:45.629035    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-8b1efeb6-5d7dd8ff94-q446n" podUID="46fa8602-cba7-4489-a2ab-57d7bdecec1a" containerName="payload-processing" containerID="cri-o://5db57f3589454d021eb4f28470fee0ac5947220c6c2e42db0014ceb2514bff7e" gracePeriod=25
Nov 03 22:35:45.686014 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:45.685972    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-8b1efeb6-68787c86c9-wpjjk" podUID="993058fc-2d6c-4c46-bf06-3aa2442b34f7" containerName="payload-pre-processing" containerID="cri-o://2e61a010a432fb082e26ef45efe0aaa669cdc1f16f330161534e97c827e96ca8" gracePeriod=25
Nov 03 22:35:45.822160 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:45.821934    4431 generic.go:356] "Generic (PLEG): container finished" podID="993058fc-2d6c-4c46-bf06-3aa2442b34f7" containerID="2e61a010a432fb082e26ef45efe0aaa669cdc1f16f330161534e97c827e96ca8" exitCode=0
Nov 03 22:35:45.822160 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:45.821998    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8b1efeb6-68787c86c9-wpjjk" event={"ID":"993058fc-2d6c-4c46-bf06-3aa2442b34f7","Type":"ContainerDied","Data":"2e61a010a432fb082e26ef45efe0aaa669cdc1f16f330161534e97c827e96ca8"}
Nov 03 22:35:45.829913 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:45.829880    4431 generic.go:356] "Generic (PLEG): container finished" podID="46fa8602-cba7-4489-a2ab-57d7bdecec1a" containerID="5db57f3589454d021eb4f28470fee0ac5947220c6c2e42db0014ceb2514bff7e" exitCode=0
Nov 03 22:35:45.830102 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:45.830023    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8b1efeb6-5d7dd8ff94-q446n" event={"ID":"46fa8602-cba7-4489-a2ab-57d7bdecec1a","Type":"ContainerDied","Data":"5db57f3589454d021eb4f28470fee0ac5947220c6c2e42db0014ceb2514bff7e"}
Nov 03 22:35:45.912692 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:45.912657    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:35:46.007495 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:46.007458    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8b1efeb6-5d7dd8ff94-q446n"
Nov 03 22:35:46.047268 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:46.047169    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8b1efeb6-68787c86c9-wpjjk"
Nov 03 22:35:46.065182 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:46.065149    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46fa8602-cba7-4489-a2ab-57d7bdecec1a-kube-api-access-x9h9s\" (UniqueName: \"kubernetes.io/projected/46fa8602-cba7-4489-a2ab-57d7bdecec1a-kube-api-access-x9h9s\") pod \"46fa8602-cba7-4489-a2ab-57d7bdecec1a\" (UID: \"46fa8602-cba7-4489-a2ab-57d7bdecec1a\") "
Nov 03 22:35:46.065406 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:46.065204    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46fa8602-cba7-4489-a2ab-57d7bdecec1a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46fa8602-cba7-4489-a2ab-57d7bdecec1a-plugins-config-volume\") pod \"46fa8602-cba7-4489-a2ab-57d7bdecec1a\" (UID: \"46fa8602-cba7-4489-a2ab-57d7bdecec1a\") "
Nov 03 22:35:46.065614 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:46.065589    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46fa8602-cba7-4489-a2ab-57d7bdecec1a-plugins-config-volume" pod "46fa8602-cba7-4489-a2ab-57d7bdecec1a" (UID: "46fa8602-cba7-4489-a2ab-57d7bdecec1a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:35:46.065996 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:46.065973    4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46fa8602-cba7-4489-a2ab-57d7bdecec1a-plugins-config-volume\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:35:46.067296 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:46.067270    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46fa8602-cba7-4489-a2ab-57d7bdecec1a-kube-api-access-x9h9s" pod "46fa8602-cba7-4489-a2ab-57d7bdecec1a" (UID: "46fa8602-cba7-4489-a2ab-57d7bdecec1a"). InnerVolumeSpecName "kube-api-access-x9h9s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:35:46.167788 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:46.167750    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/993058fc-2d6c-4c46-bf06-3aa2442b34f7-kube-api-access-g9rcp\" (UniqueName: \"kubernetes.io/projected/993058fc-2d6c-4c46-bf06-3aa2442b34f7-kube-api-access-g9rcp\") pod \"993058fc-2d6c-4c46-bf06-3aa2442b34f7\" (UID: \"993058fc-2d6c-4c46-bf06-3aa2442b34f7\") "
Nov 03 22:35:46.167997 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:46.167865    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/993058fc-2d6c-4c46-bf06-3aa2442b34f7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/993058fc-2d6c-4c46-bf06-3aa2442b34f7-plugins-config-volume\") pod \"993058fc-2d6c-4c46-bf06-3aa2442b34f7\" (UID: \"993058fc-2d6c-4c46-bf06-3aa2442b34f7\") "
Nov 03 22:35:46.168959 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:46.168371    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/993058fc-2d6c-4c46-bf06-3aa2442b34f7-plugins-config-volume" pod "993058fc-2d6c-4c46-bf06-3aa2442b34f7" (UID: "993058fc-2d6c-4c46-bf06-3aa2442b34f7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:35:46.168959 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:46.168636    4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/993058fc-2d6c-4c46-bf06-3aa2442b34f7-plugins-config-volume\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:35:46.168959 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:46.168660    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9h9s\" (UniqueName: \"kubernetes.io/projected/46fa8602-cba7-4489-a2ab-57d7bdecec1a-kube-api-access-x9h9s\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:35:46.170811 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:46.170784    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/993058fc-2d6c-4c46-bf06-3aa2442b34f7-kube-api-access-g9rcp" pod "993058fc-2d6c-4c46-bf06-3aa2442b34f7" (UID: "993058fc-2d6c-4c46-bf06-3aa2442b34f7"). InnerVolumeSpecName "kube-api-access-g9rcp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:35:46.269968 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:46.269928    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9rcp\" (UniqueName: \"kubernetes.io/projected/993058fc-2d6c-4c46-bf06-3aa2442b34f7-kube-api-access-g9rcp\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:35:46.309729 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:46.309655    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:35:46.813358 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:46.813324    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:35:46.838059 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:46.838020    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8b1efeb6-5d7dd8ff94-q446n"
Nov 03 22:35:46.838277 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:46.838072    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8b1efeb6-5d7dd8ff94-q446n" event={"ID":"46fa8602-cba7-4489-a2ab-57d7bdecec1a","Type":"ContainerDied","Data":"fbbe0ed204f348067b2037417961359e204d23b1d4b9649038daae422583357f"}
Nov 03 22:35:46.838277 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:46.838136    4431 scope.go:122] "RemoveContainer" containerID="5db57f3589454d021eb4f28470fee0ac5947220c6c2e42db0014ceb2514bff7e"
Nov 03 22:35:46.839786 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:46.839765    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8b1efeb6-68787c86c9-wpjjk"
Nov 03 22:35:46.839884 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:46.839790    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8b1efeb6-68787c86c9-wpjjk" event={"ID":"993058fc-2d6c-4c46-bf06-3aa2442b34f7","Type":"ContainerDied","Data":"44bcc8dd1036284f2047b965a74216005f8ba54b6e9d1bdbfd121b25832c19ad"}
Nov 03 22:35:46.849857 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:46.849841    4431 scope.go:122] "RemoveContainer" containerID="2e61a010a432fb082e26ef45efe0aaa669cdc1f16f330161534e97c827e96ca8"
Nov 03 22:35:46.897345 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:46.897316    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8b1efeb6-68787c86c9-wpjjk"]
Nov 03 22:35:46.908756 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:46.908682    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8b1efeb6-68787c86c9-wpjjk"]
Nov 03 22:35:46.936618 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:46.936590    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8b1efeb6-5d7dd8ff94-q446n"]
Nov 03 22:35:46.940475 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:46.940450    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8b1efeb6-5d7dd8ff94-q446n"]
Nov 03 22:35:47.735883 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:47.735827    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:35:47.900480 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:47.900449    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:35:48.102051 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:48.101950    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:35:48.207016 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:48.206829    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46fa8602-cba7-4489-a2ab-57d7bdecec1a" path="/var/lib/kubelet/pods/46fa8602-cba7-4489-a2ab-57d7bdecec1a/volumes"
Nov 03 22:35:48.208188 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:48.208154    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="993058fc-2d6c-4c46-bf06-3aa2442b34f7" path="/var/lib/kubelet/pods/993058fc-2d6c-4c46-bf06-3aa2442b34f7/volumes"
Nov 03 22:35:49.691528 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:49.691458    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:35:51.333253 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:51.333159    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:35:51.847864 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:51.847832    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:35:53.132564 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:53.132528    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:35:54.834796 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:54.834758    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:35:56.592731 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:56.592678    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:35:58.031336 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:58.031290    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:35:59.768167 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:35:59.768128    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:36:01.260929 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:36:01.260890    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:36:01.410317 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:01.410284    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:36:03.045809 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:03.045766    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:36:04.807645 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:04.807609    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:36:06.654027 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:06.653987    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:36:08.133875 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:08.131052    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:36:09.676130 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:09.676090    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:36:10.360600 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:10.360553    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:36:11.009855 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:11.009825    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:36:11.399384 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:11.399288    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:36:11.968935 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:11.968894    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:36:12.175560 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:12.175227    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:36:12.336030 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:12.335943    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:36:13.444640 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:13.444601    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:36:13.598904 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:13.598867    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:36:14.105839 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:14.105756    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:36:17.537330 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:17.537272    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:36:17.728834 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:17.728794    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:36:19.772154 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:19.771352    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:36:21.401684 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:21.401651    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:36:23.039573 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:23.039534    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:36:23.183731 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:23.183698    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:36:23.339939 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:23.339854    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:36:24.211969 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:24.211933    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4c5345-5548f954c-kk85c"]
Nov 03 22:36:24.429492 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:24.429460    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:36:24.593295 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:24.593186    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:36:24.826683 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:24.826651    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:36:26.553270 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:26.550725    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:36:28.115573 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:28.115538    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:36:28.324122 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:28.324089    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:36:28.572380 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:28.572348    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:36:29.184063 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:29.184021    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:36:29.213153 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:29.213110    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4c5345-5548f954c-kk85c" podUID="07bc79a3-84e3-4852-a5cb-6cd9c0d9eceb" containerName="payload-pre-processing" containerID="cri-o://1d0fb38f1f0ff2fc1653daad57a782510ce80de9122f4a75d0eb2dffb16555fe" gracePeriod=25
Nov 03 22:36:29.491946 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:29.491919    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4c5345-5548f954c-kk85c"
Nov 03 22:36:29.551018 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:29.550985    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07bc79a3-84e3-4852-a5cb-6cd9c0d9eceb-kube-api-access-g7c75\" (UniqueName: \"kubernetes.io/projected/07bc79a3-84e3-4852-a5cb-6cd9c0d9eceb-kube-api-access-g7c75\") pod \"07bc79a3-84e3-4852-a5cb-6cd9c0d9eceb\" (UID: \"07bc79a3-84e3-4852-a5cb-6cd9c0d9eceb\") "
Nov 03 22:36:29.551241 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:29.551062    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07bc79a3-84e3-4852-a5cb-6cd9c0d9eceb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07bc79a3-84e3-4852-a5cb-6cd9c0d9eceb-plugins-config-volume\") pod \"07bc79a3-84e3-4852-a5cb-6cd9c0d9eceb\" (UID: \"07bc79a3-84e3-4852-a5cb-6cd9c0d9eceb\") "
Nov 03 22:36:29.551572 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:29.551542    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07bc79a3-84e3-4852-a5cb-6cd9c0d9eceb-plugins-config-volume" pod "07bc79a3-84e3-4852-a5cb-6cd9c0d9eceb" (UID: "07bc79a3-84e3-4852-a5cb-6cd9c0d9eceb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:36:29.552017 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:29.551993    4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07bc79a3-84e3-4852-a5cb-6cd9c0d9eceb-plugins-config-volume\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:36:29.553471 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:29.553449    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07bc79a3-84e3-4852-a5cb-6cd9c0d9eceb-kube-api-access-g7c75" pod "07bc79a3-84e3-4852-a5cb-6cd9c0d9eceb" (UID: "07bc79a3-84e3-4852-a5cb-6cd9c0d9eceb"). InnerVolumeSpecName "kube-api-access-g7c75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:36:29.653873 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:29.653842    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7c75\" (UniqueName: \"kubernetes.io/projected/07bc79a3-84e3-4852-a5cb-6cd9c0d9eceb-kube-api-access-g7c75\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:36:29.805326 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:29.805244    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:36:30.294312 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:30.294274    4431 generic.go:356] "Generic (PLEG): container finished" podID="07bc79a3-84e3-4852-a5cb-6cd9c0d9eceb" containerID="1d0fb38f1f0ff2fc1653daad57a782510ce80de9122f4a75d0eb2dffb16555fe" exitCode=0
Nov 03 22:36:30.294312 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:30.294309    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4c5345-5548f954c-kk85c"
Nov 03 22:36:30.294890 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:30.294308    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4c5345-5548f954c-kk85c" event={"ID":"07bc79a3-84e3-4852-a5cb-6cd9c0d9eceb","Type":"ContainerDied","Data":"1d0fb38f1f0ff2fc1653daad57a782510ce80de9122f4a75d0eb2dffb16555fe"}
Nov 03 22:36:30.294890 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:30.294441    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4c5345-5548f954c-kk85c" event={"ID":"07bc79a3-84e3-4852-a5cb-6cd9c0d9eceb","Type":"ContainerDied","Data":"d70545080ff3579395ba09d30d23bb9a63d4642354397cb614832ed5de67ac7f"}
Nov 03 22:36:30.294890 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:30.294483    4431 scope.go:122] "RemoveContainer" containerID="1d0fb38f1f0ff2fc1653daad57a782510ce80de9122f4a75d0eb2dffb16555fe"
Nov 03 22:36:30.305984 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:30.305956    4431 scope.go:122] "RemoveContainer" containerID="1d0fb38f1f0ff2fc1653daad57a782510ce80de9122f4a75d0eb2dffb16555fe"
Nov 03 22:36:30.306315 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:36:30.306292    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d0fb38f1f0ff2fc1653daad57a782510ce80de9122f4a75d0eb2dffb16555fe\": container with ID starting with 1d0fb38f1f0ff2fc1653daad57a782510ce80de9122f4a75d0eb2dffb16555fe not found: ID does not exist" containerID="1d0fb38f1f0ff2fc1653daad57a782510ce80de9122f4a75d0eb2dffb16555fe"
Nov 03 22:36:30.306415 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:30.306325    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d0fb38f1f0ff2fc1653daad57a782510ce80de9122f4a75d0eb2dffb16555fe"} err="failed to get container status \"1d0fb38f1f0ff2fc1653daad57a782510ce80de9122f4a75d0eb2dffb16555fe\": rpc error: code = NotFound desc = could not find container \"1d0fb38f1f0ff2fc1653daad57a782510ce80de9122f4a75d0eb2dffb16555fe\": container with ID starting with 1d0fb38f1f0ff2fc1653daad57a782510ce80de9122f4a75d0eb2dffb16555fe not found: ID does not exist"
Nov 03 22:36:30.325929 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:30.325900    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:36:30.362209 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:30.362131    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4c5345-5548f954c-kk85c"]
Nov 03 22:36:30.373440 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:30.373408    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4c5345-5548f954c-kk85c"]
Nov 03 22:36:30.540242 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:30.531860    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:36:31.261756 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:36:31.261715    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:36:31.926881 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:31.926823    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:36:32.110535 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:32.110498    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07bc79a3-84e3-4852-a5cb-6cd9c0d9eceb" path="/var/lib/kubelet/pods/07bc79a3-84e3-4852-a5cb-6cd9c0d9eceb/volumes"
Nov 03 22:36:32.450336 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:32.450272    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:36:32.597780 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:32.597689    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:36:34.167367 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:34.167327    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:36:34.359102 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:34.359052    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:36:34.590638 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:34.590546    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:36:35.510313 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:35.510281    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:36:35.654737 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:35.654695    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:36:36.571087 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:36.571051    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:36:36.721202 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:36.721171    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:36:38.275320 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:38.275287    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:36:39.925031 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:39.924996    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:36:40.136624 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:40.132727    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:36:40.426464 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:40.426400    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:36:40.978232 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:40.976320    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:36:41.187931 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:41.186398    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:36:42.611538 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:42.611477    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:36:44.231575 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:44.231523    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:36:45.725862 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:45.725826    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:36:46.923941 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:46.923845    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:36:47.336360 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:47.330743    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:36:48.938264 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:48.938205    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:36:49.554320 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:49.554284    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:36:49.612383 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:49.612349    4431 generic.go:356] "Generic (PLEG): container finished" podID="10b331d4-81e4-4243-a645-69d95958db00" containerID="9f24b059f48447fac205ac83277f336c7be557b4493c31260420aa82e575e556" exitCode=0
Nov 03 22:36:49.612525 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:49.612394    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss" event={"ID":"10b331d4-81e4-4243-a645-69d95958db00","Type":"ContainerDied","Data":"9f24b059f48447fac205ac83277f336c7be557b4493c31260420aa82e575e556"}
Nov 03 22:36:49.612876 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:49.612859    4431 scope.go:122] "RemoveContainer" containerID="9f24b059f48447fac205ac83277f336c7be557b4493c31260420aa82e575e556"
Nov 03 22:36:49.614846 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:49.614818    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:36:49.614930 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:49.614913    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:36:49.614968 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:49.614948    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:36:50.605826 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:50.605781    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:36:50.687574 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:50.687536    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss" event={"ID":"10b331d4-81e4-4243-a645-69d95958db00","Type":"ContainerStarted","Data":"01351340e51ce0fb93b3c787cb2f6052ae23c02b50a12649390f2aaee60d4b4f"}
Nov 03 22:36:50.709268 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:50.709206    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss"
Nov 03 22:36:50.710503 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:50.710469    4431 patch_prober.go:29] interesting pod/e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Nov 03 22:36:50.710652 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:50.710534    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss" podUID="10b331d4-81e4-4243-a645-69d95958db00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Nov 03 22:36:50.726624 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:50.726596    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss"
Nov 03 22:36:51.710294 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:51.710263    4431 patch_prober.go:29] interesting pod/e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Nov 03 22:36:51.710719 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:51.710327    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss" podUID="10b331d4-81e4-4243-a645-69d95958db00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Nov 03 22:36:51.974298 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:51.968089    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:36:52.223006 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:52.222950    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:36:52.710107 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:52.710074    4431 patch_prober.go:29] interesting pod/e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Nov 03 22:36:52.710301 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:52.710135    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss" podUID="10b331d4-81e4-4243-a645-69d95958db00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Nov 03 22:36:53.783462 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:53.783108    4431 patch_prober.go:29] interesting pod/e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Nov 03 22:36:53.783462 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:53.783181    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss" podUID="10b331d4-81e4-4243-a645-69d95958db00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Nov 03 22:36:53.804252 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:53.794322    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:36:53.941242 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:53.939826    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:36:54.383691 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:54.383652    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:36:54.712239 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:54.711192    4431 patch_prober.go:29] interesting pod/e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Nov 03 22:36:54.712239 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:54.711268    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss" podUID="10b331d4-81e4-4243-a645-69d95958db00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Nov 03 22:36:55.687237 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:55.686727    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:36:55.709887 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:55.709845    4431 patch_prober.go:29] interesting pod/e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Nov 03 22:36:55.710097 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:55.709931    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss" podUID="10b331d4-81e4-4243-a645-69d95958db00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Nov 03 22:36:56.717102 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:56.717069    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss"
Nov 03 22:36:56.718332 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:56.718305    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss"
Nov 03 22:36:57.168074 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:57.168040    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:36:57.312499 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:57.312468    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:36:58.205456 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:58.205429    4431 scope.go:122] "RemoveContainer" containerID="c8018975d1bd8c374957081ee571b57d9bb9feef7ccdf6c40bd811c38bce8661"
Nov 03 22:36:58.883297 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:58.883260    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:36:59.039171 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:59.039118    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:36:59.458751 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:59.458710    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:36:59.905903 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:36:59.905825    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:37:00.367773 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:00.367730    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:37:00.627602 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:00.627493    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:37:01.261735 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:37:01.261698    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:37:02.228268 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:02.228230    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:37:03.924649 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:03.924613    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:37:04.244233 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:04.244139    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:37:04.502681 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:04.502591    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:37:05.981772 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:05.981736    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:37:06.036000 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:06.035968    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2d6076-d576997dd-rh48p"]
Nov 03 22:37:06.036722 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:06.036706    4431 cpu_manager.go:414] "containerMap: removing container" podUID="46fa8602-cba7-4489-a2ab-57d7bdecec1a" containerName="payload-processing"
Nov 03 22:37:06.036787 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:06.036725    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46fa8602-cba7-4489-a2ab-57d7bdecec1a" containerName="payload-processing"
Nov 03 22:37:06.036787 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:06.036744    4431 cpu_manager.go:414] "containerMap: removing container" podUID="993058fc-2d6c-4c46-bf06-3aa2442b34f7" containerName="payload-pre-processing"
Nov 03 22:37:06.036787 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:06.036749    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="993058fc-2d6c-4c46-bf06-3aa2442b34f7" containerName="payload-pre-processing"
Nov 03 22:37:06.036888 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:06.036793    4431 cpu_manager.go:414] "containerMap: removing container" podUID="07bc79a3-84e3-4852-a5cb-6cd9c0d9eceb" containerName="payload-pre-processing"
Nov 03 22:37:06.036888 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:06.036800    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07bc79a3-84e3-4852-a5cb-6cd9c0d9eceb" containerName="payload-pre-processing"
Nov 03 22:37:06.036953 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:06.036909    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="46fa8602-cba7-4489-a2ab-57d7bdecec1a" containerName="payload-processing"
Nov 03 22:37:06.036953 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:06.036919    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="993058fc-2d6c-4c46-bf06-3aa2442b34f7" containerName="payload-pre-processing"
Nov 03 22:37:06.036953 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:06.036933    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="07bc79a3-84e3-4852-a5cb-6cd9c0d9eceb" containerName="payload-pre-processing"
Nov 03 22:37:06.040684 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:06.040661    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2d6076-d576997dd-rh48p"
Nov 03 22:37:06.063672 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:06.063642    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2d6076-d576997dd-rh48p"]
Nov 03 22:37:06.078256 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:06.070419    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-2d6076\""
Nov 03 22:37:06.120754 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:06.120715    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jqff\" (UniqueName: \"kubernetes.io/projected/9749991e-8f89-488c-9f71-e7da0c122683-kube-api-access-4jqff\") pod \"payload-processing-e2e-inf-b-2d6076-d576997dd-rh48p\" (UID: \"9749991e-8f89-488c-9f71-e7da0c122683\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2d6076-d576997dd-rh48p"
Nov 03 22:37:06.120988 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:06.120797    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9749991e-8f89-488c-9f71-e7da0c122683-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-2d6076-d576997dd-rh48p\" (UID: \"9749991e-8f89-488c-9f71-e7da0c122683\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2d6076-d576997dd-rh48p"
Nov 03 22:37:06.148911 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:06.148883    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:37:06.241368 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:06.241277    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jqff\" (UniqueName: \"kubernetes.io/projected/9749991e-8f89-488c-9f71-e7da0c122683-kube-api-access-4jqff\") pod \"payload-processing-e2e-inf-b-2d6076-d576997dd-rh48p\" (UID: \"9749991e-8f89-488c-9f71-e7da0c122683\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2d6076-d576997dd-rh48p"
Nov 03 22:37:06.241638 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:06.241614    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9749991e-8f89-488c-9f71-e7da0c122683-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-2d6076-d576997dd-rh48p\" (UID: \"9749991e-8f89-488c-9f71-e7da0c122683\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2d6076-d576997dd-rh48p"
Nov 03 22:37:06.253975 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:06.242986    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9749991e-8f89-488c-9f71-e7da0c122683-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-2d6076-d576997dd-rh48p\" (UID: \"9749991e-8f89-488c-9f71-e7da0c122683\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2d6076-d576997dd-rh48p"
Nov 03 22:37:06.310704 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:06.310667    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:37:06.311782 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:06.311753    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jqff\" (UniqueName: \"kubernetes.io/projected/9749991e-8f89-488c-9f71-e7da0c122683-kube-api-access-4jqff\") pod \"payload-processing-e2e-inf-b-2d6076-d576997dd-rh48p\" (UID: \"9749991e-8f89-488c-9f71-e7da0c122683\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2d6076-d576997dd-rh48p"
Nov 03 22:37:06.367306 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:06.367267    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2d6076-d576997dd-rh48p"
Nov 03 22:37:06.793398 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:06.793371    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2d6076-d576997dd-rh48p"]
Nov 03 22:37:06.922690 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:06.922658    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2d6076-d576997dd-rh48p" event={"ID":"9749991e-8f89-488c-9f71-e7da0c122683","Type":"ContainerStarted","Data":"d74ce5a66b2142027aab3d1b95727352ee5af095ad4c4cca92e9db6559d670e9"}
Nov 03 22:37:06.922860 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:06.922713    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2d6076-d576997dd-rh48p" event={"ID":"9749991e-8f89-488c-9f71-e7da0c122683","Type":"ContainerStarted","Data":"0df831cd754b8f01cc4ea5fd74f3d027624e305c0c6850dd130639a9f88bd86a"}
Nov 03 22:37:06.922860 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:06.922753    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-2d6076-d576997dd-rh48p"
Nov 03 22:37:06.963755 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:06.963703    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-2d6076-d576997dd-rh48p" podStartSLOduration=0.963684916 podStartE2EDuration="963.684916ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:37:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:37:06.963078311 +0000 UTC m=+1450.988934173" watchObservedRunningTime="2026-11-03 22:37:06.963684916 +0000 UTC m=+1450.989540782"
Nov 03 22:37:07.615007 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:07.614972    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:37:07.999248 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:07.999187    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:37:08.257001 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:08.256916    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:37:08.722082 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:08.722050    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:37:09.571075 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:09.571035    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:37:09.716278 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:09.716197    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:37:09.997795 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:09.997757    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7"]
Nov 03 22:37:09.998383 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:09.997983    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7" podUID="6634b1a8-4b77-4281-8566-3470ecf3022f" containerName="istio-proxy" containerID="cri-o://8c9ec34de308f6be5a9d4ca11032917bbe1543513b0793d8895af9374c1d92ff" gracePeriod=30
Nov 03 22:37:11.463300 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:11.463265    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8b1efeb6-98884dc8c-v2chc"]
Nov 03 22:37:11.466853 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:11.466831    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8b1efeb6-98884dc8c-v2chc"
Nov 03 22:37:11.496150 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:11.496107    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-8b1efeb6-serving-cert\""
Nov 03 22:37:11.503850 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:11.503820    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8b1efeb6-98884dc8c-v2chc"]
Nov 03 22:37:11.583547 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:11.583512    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/058f82b5-15b1-4757-8cc9-fcf85f287ef3-openshift-service-ca\") pod \"maas-api-e2e-mt-8b1efeb6-98884dc8c-v2chc\" (UID: \"058f82b5-15b1-4757-8cc9-fcf85f287ef3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8b1efeb6-98884dc8c-v2chc"
Nov 03 22:37:11.583761 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:11.583559    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/058f82b5-15b1-4757-8cc9-fcf85f287ef3-maas-api-tls\") pod \"maas-api-e2e-mt-8b1efeb6-98884dc8c-v2chc\" (UID: \"058f82b5-15b1-4757-8cc9-fcf85f287ef3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8b1efeb6-98884dc8c-v2chc"
Nov 03 22:37:11.583761 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:11.583662    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/058f82b5-15b1-4757-8cc9-fcf85f287ef3-metrics-certs\") pod \"maas-api-e2e-mt-8b1efeb6-98884dc8c-v2chc\" (UID: \"058f82b5-15b1-4757-8cc9-fcf85f287ef3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8b1efeb6-98884dc8c-v2chc"
Nov 03 22:37:11.583761 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:11.583740    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l854b\" (UniqueName: \"kubernetes.io/projected/058f82b5-15b1-4757-8cc9-fcf85f287ef3-kube-api-access-l854b\") pod \"maas-api-e2e-mt-8b1efeb6-98884dc8c-v2chc\" (UID: \"058f82b5-15b1-4757-8cc9-fcf85f287ef3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8b1efeb6-98884dc8c-v2chc"
Nov 03 22:37:11.662438 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:11.662403    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:37:11.685299 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:11.685273    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/058f82b5-15b1-4757-8cc9-fcf85f287ef3-openshift-service-ca\") pod \"maas-api-e2e-mt-8b1efeb6-98884dc8c-v2chc\" (UID: \"058f82b5-15b1-4757-8cc9-fcf85f287ef3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8b1efeb6-98884dc8c-v2chc"
Nov 03 22:37:11.685466 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:11.685315    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/058f82b5-15b1-4757-8cc9-fcf85f287ef3-maas-api-tls\") pod \"maas-api-e2e-mt-8b1efeb6-98884dc8c-v2chc\" (UID: \"058f82b5-15b1-4757-8cc9-fcf85f287ef3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8b1efeb6-98884dc8c-v2chc"
Nov 03 22:37:11.685466 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:11.685429    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/058f82b5-15b1-4757-8cc9-fcf85f287ef3-metrics-certs\") pod \"maas-api-e2e-mt-8b1efeb6-98884dc8c-v2chc\" (UID: \"058f82b5-15b1-4757-8cc9-fcf85f287ef3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8b1efeb6-98884dc8c-v2chc"
Nov 03 22:37:11.685579 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:11.685491    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l854b\" (UniqueName: \"kubernetes.io/projected/058f82b5-15b1-4757-8cc9-fcf85f287ef3-kube-api-access-l854b\") pod \"maas-api-e2e-mt-8b1efeb6-98884dc8c-v2chc\" (UID: \"058f82b5-15b1-4757-8cc9-fcf85f287ef3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8b1efeb6-98884dc8c-v2chc"
Nov 03 22:37:11.686045 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:11.686006    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/058f82b5-15b1-4757-8cc9-fcf85f287ef3-openshift-service-ca\") pod \"maas-api-e2e-mt-8b1efeb6-98884dc8c-v2chc\" (UID: \"058f82b5-15b1-4757-8cc9-fcf85f287ef3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8b1efeb6-98884dc8c-v2chc"
Nov 03 22:37:11.687901 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:11.687883    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/058f82b5-15b1-4757-8cc9-fcf85f287ef3-maas-api-tls\") pod \"maas-api-e2e-mt-8b1efeb6-98884dc8c-v2chc\" (UID: \"058f82b5-15b1-4757-8cc9-fcf85f287ef3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8b1efeb6-98884dc8c-v2chc"
Nov 03 22:37:11.687999 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:11.687983    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/058f82b5-15b1-4757-8cc9-fcf85f287ef3-metrics-certs\") pod \"maas-api-e2e-mt-8b1efeb6-98884dc8c-v2chc\" (UID: \"058f82b5-15b1-4757-8cc9-fcf85f287ef3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8b1efeb6-98884dc8c-v2chc"
Nov 03 22:37:11.709182 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:11.709155    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l854b\" (UniqueName: \"kubernetes.io/projected/058f82b5-15b1-4757-8cc9-fcf85f287ef3-kube-api-access-l854b\") pod \"maas-api-e2e-mt-8b1efeb6-98884dc8c-v2chc\" (UID: \"058f82b5-15b1-4757-8cc9-fcf85f287ef3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8b1efeb6-98884dc8c-v2chc"
Nov 03 22:37:11.781014 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:11.780918    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8b1efeb6-98884dc8c-v2chc"
Nov 03 22:37:11.832522 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:11.829873    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:37:12.114712 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:12.114676    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8b1efeb6-98884dc8c-v2chc"]
Nov 03 22:37:12.114860 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:37:12.114773    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod058f82b5_15b1_4757_8cc9_fcf85f287ef3.slice/crio-3dcaa32a148277fe4054c96a9d1db9689bf091d8acbf4086f6b6adf17e73a44d WatchSource:0}: Error finding container 3dcaa32a148277fe4054c96a9d1db9689bf091d8acbf4086f6b6adf17e73a44d: Status 404 returned error can't find the container with id 3dcaa32a148277fe4054c96a9d1db9689bf091d8acbf4086f6b6adf17e73a44d
Nov 03 22:37:12.716556 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:12.716464    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:37:12.890019 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:12.889988    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:37:13.006553 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:13.006453    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8b1efeb6-98884dc8c-v2chc" event={"ID":"058f82b5-15b1-4757-8cc9-fcf85f287ef3","Type":"ContainerStarted","Data":"cbf63fc7373cc42648112d74ccec6986216a33a37ce6053a512cd012205ee8bf"}
Nov 03 22:37:13.006553 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:13.006514    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8b1efeb6-98884dc8c-v2chc" event={"ID":"058f82b5-15b1-4757-8cc9-fcf85f287ef3","Type":"ContainerStarted","Data":"3dcaa32a148277fe4054c96a9d1db9689bf091d8acbf4086f6b6adf17e73a44d"}
Nov 03 22:37:13.006553 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:13.006539    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8b1efeb6-98884dc8c-v2chc"
Nov 03 22:37:13.064750 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:13.064698    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8b1efeb6-98884dc8c-v2chc" podStartSLOduration=1.7705084169999998 podStartE2EDuration="2.064683056s" totalImagesPullingTime="294.174639ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:37:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:37:13.061297924 +0000 UTC m=+1457.087153790" watchObservedRunningTime="2026-11-03 22:37:13.064683056 +0000 UTC m=+1457.090538917"
Nov 03 22:37:13.487099 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:13.487058    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:37:13.657542 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:13.657510    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:37:13.853310 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:13.853042    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:37:15.302831 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:15.302597    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:37:15.343010 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:15.342984    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7"
Nov 03 22:37:15.460135 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:15.460086    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6634b1a8-4b77-4281-8566-3470ecf3022f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6634b1a8-4b77-4281-8566-3470ecf3022f-workload-socket\") pod \"6634b1a8-4b77-4281-8566-3470ecf3022f\" (UID: \"6634b1a8-4b77-4281-8566-3470ecf3022f\") "
Nov 03 22:37:15.460338 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:15.460318    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6634b1a8-4b77-4281-8566-3470ecf3022f-istio-token\" (UniqueName: \"kubernetes.io/projected/6634b1a8-4b77-4281-8566-3470ecf3022f-istio-token\") pod \"6634b1a8-4b77-4281-8566-3470ecf3022f\" (UID: \"6634b1a8-4b77-4281-8566-3470ecf3022f\") "
Nov 03 22:37:15.460395 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:15.460360    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6634b1a8-4b77-4281-8566-3470ecf3022f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6634b1a8-4b77-4281-8566-3470ecf3022f-istio-podinfo\") pod \"6634b1a8-4b77-4281-8566-3470ecf3022f\" (UID: \"6634b1a8-4b77-4281-8566-3470ecf3022f\") "
Nov 03 22:37:15.460436 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:15.460402    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6634b1a8-4b77-4281-8566-3470ecf3022f-workload-socket" pod "6634b1a8-4b77-4281-8566-3470ecf3022f" (UID: "6634b1a8-4b77-4281-8566-3470ecf3022f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:37:15.460478 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:15.460449    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6634b1a8-4b77-4281-8566-3470ecf3022f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6634b1a8-4b77-4281-8566-3470ecf3022f-istiod-ca-cert\") pod \"6634b1a8-4b77-4281-8566-3470ecf3022f\" (UID: \"6634b1a8-4b77-4281-8566-3470ecf3022f\") "
Nov 03 22:37:15.460524 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:15.460481    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6634b1a8-4b77-4281-8566-3470ecf3022f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6634b1a8-4b77-4281-8566-3470ecf3022f-istio-envoy\") pod \"6634b1a8-4b77-4281-8566-3470ecf3022f\" (UID: \"6634b1a8-4b77-4281-8566-3470ecf3022f\") "
Nov 03 22:37:15.460524 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:15.460512    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6634b1a8-4b77-4281-8566-3470ecf3022f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6634b1a8-4b77-4281-8566-3470ecf3022f-workload-certs\") pod \"6634b1a8-4b77-4281-8566-3470ecf3022f\" (UID: \"6634b1a8-4b77-4281-8566-3470ecf3022f\") "
Nov 03 22:37:15.460651 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:15.460566    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6634b1a8-4b77-4281-8566-3470ecf3022f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6634b1a8-4b77-4281-8566-3470ecf3022f-istio-data\") pod \"6634b1a8-4b77-4281-8566-3470ecf3022f\" (UID: \"6634b1a8-4b77-4281-8566-3470ecf3022f\") "
Nov 03 22:37:15.460708 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:15.460675    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6634b1a8-4b77-4281-8566-3470ecf3022f-kube-api-access-jj95d\" (UniqueName: \"kubernetes.io/projected/6634b1a8-4b77-4281-8566-3470ecf3022f-kube-api-access-jj95d\") pod \"6634b1a8-4b77-4281-8566-3470ecf3022f\" (UID: \"6634b1a8-4b77-4281-8566-3470ecf3022f\") "
Nov 03 22:37:15.460811 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:15.460785    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6634b1a8-4b77-4281-8566-3470ecf3022f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6634b1a8-4b77-4281-8566-3470ecf3022f-credential-socket\") pod \"6634b1a8-4b77-4281-8566-3470ecf3022f\" (UID: \"6634b1a8-4b77-4281-8566-3470ecf3022f\") "
Nov 03 22:37:15.460811 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:15.460798    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6634b1a8-4b77-4281-8566-3470ecf3022f-workload-certs" pod "6634b1a8-4b77-4281-8566-3470ecf3022f" (UID: "6634b1a8-4b77-4281-8566-3470ecf3022f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:37:15.461007 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:15.460885    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6634b1a8-4b77-4281-8566-3470ecf3022f-istiod-ca-cert" pod "6634b1a8-4b77-4281-8566-3470ecf3022f" (UID: "6634b1a8-4b77-4281-8566-3470ecf3022f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:37:15.461007 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:15.460908    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6634b1a8-4b77-4281-8566-3470ecf3022f-istio-data" pod "6634b1a8-4b77-4281-8566-3470ecf3022f" (UID: "6634b1a8-4b77-4281-8566-3470ecf3022f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:37:15.461127 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:15.461095    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6634b1a8-4b77-4281-8566-3470ecf3022f-credential-socket" pod "6634b1a8-4b77-4281-8566-3470ecf3022f" (UID: "6634b1a8-4b77-4281-8566-3470ecf3022f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:37:15.461728 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:15.461703    4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6634b1a8-4b77-4281-8566-3470ecf3022f-istiod-ca-cert\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:37:15.461728 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:15.461727    4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6634b1a8-4b77-4281-8566-3470ecf3022f-workload-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:37:15.461889 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:15.461736    4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6634b1a8-4b77-4281-8566-3470ecf3022f-istio-data\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:37:15.461889 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:15.461744    4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6634b1a8-4b77-4281-8566-3470ecf3022f-credential-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:37:15.461889 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:15.461755    4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6634b1a8-4b77-4281-8566-3470ecf3022f-workload-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:37:15.463128 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:15.463102    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6634b1a8-4b77-4281-8566-3470ecf3022f-istio-podinfo" pod "6634b1a8-4b77-4281-8566-3470ecf3022f" (UID: "6634b1a8-4b77-4281-8566-3470ecf3022f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 22:37:15.463249 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:15.463103    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6634b1a8-4b77-4281-8566-3470ecf3022f-istio-envoy" pod "6634b1a8-4b77-4281-8566-3470ecf3022f" (UID: "6634b1a8-4b77-4281-8566-3470ecf3022f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:37:15.463482 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:15.463464    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6634b1a8-4b77-4281-8566-3470ecf3022f-kube-api-access-jj95d" pod "6634b1a8-4b77-4281-8566-3470ecf3022f" (UID: "6634b1a8-4b77-4281-8566-3470ecf3022f"). InnerVolumeSpecName "kube-api-access-jj95d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:37:15.463555 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:15.463506    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6634b1a8-4b77-4281-8566-3470ecf3022f-istio-token" pod "6634b1a8-4b77-4281-8566-3470ecf3022f" (UID: "6634b1a8-4b77-4281-8566-3470ecf3022f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:37:15.563316 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:15.563208    4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6634b1a8-4b77-4281-8566-3470ecf3022f-istio-token\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:37:15.563316 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:15.563268    4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6634b1a8-4b77-4281-8566-3470ecf3022f-istio-podinfo\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:37:15.563316 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:15.563278    4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6634b1a8-4b77-4281-8566-3470ecf3022f-istio-envoy\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:37:15.563316 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:15.563286    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jj95d\" (UniqueName: \"kubernetes.io/projected/6634b1a8-4b77-4281-8566-3470ecf3022f-kube-api-access-jj95d\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:37:15.870524 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:15.870411    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:37:16.065356 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:16.065309    4431 generic.go:356] "Generic (PLEG): container finished" podID="6634b1a8-4b77-4281-8566-3470ecf3022f" containerID="8c9ec34de308f6be5a9d4ca11032917bbe1543513b0793d8895af9374c1d92ff" exitCode=0
Nov 03 22:37:16.065571 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:16.065547    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7" event={"ID":"6634b1a8-4b77-4281-8566-3470ecf3022f","Type":"ContainerDied","Data":"8c9ec34de308f6be5a9d4ca11032917bbe1543513b0793d8895af9374c1d92ff"}
Nov 03 22:37:16.065642 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:16.065615    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7" event={"ID":"6634b1a8-4b77-4281-8566-3470ecf3022f","Type":"ContainerDied","Data":"43f6bb2e6160edf20256c4013cf4ca89f0c2990057b0417ad413370f7cb4c7b5"}
Nov 03 22:37:16.065708 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:16.065646    4431 scope.go:122] "RemoveContainer" containerID="8c9ec34de308f6be5a9d4ca11032917bbe1543513b0793d8895af9374c1d92ff"
Nov 03 22:37:16.065871 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:16.065857    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7"
Nov 03 22:37:16.231301 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:16.219098    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:37:16.271960 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:16.271932    4431 scope.go:122] "RemoveContainer" containerID="8c9ec34de308f6be5a9d4ca11032917bbe1543513b0793d8895af9374c1d92ff"
Nov 03 22:37:16.272436 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:37:16.272412    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c9ec34de308f6be5a9d4ca11032917bbe1543513b0793d8895af9374c1d92ff\": container with ID starting with 8c9ec34de308f6be5a9d4ca11032917bbe1543513b0793d8895af9374c1d92ff not found: ID does not exist" containerID="8c9ec34de308f6be5a9d4ca11032917bbe1543513b0793d8895af9374c1d92ff"
Nov 03 22:37:16.272530 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:16.272447    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c9ec34de308f6be5a9d4ca11032917bbe1543513b0793d8895af9374c1d92ff"} err="failed to get container status \"8c9ec34de308f6be5a9d4ca11032917bbe1543513b0793d8895af9374c1d92ff\": rpc error: code = NotFound desc = could not find container \"8c9ec34de308f6be5a9d4ca11032917bbe1543513b0793d8895af9374c1d92ff\": container with ID starting with 8c9ec34de308f6be5a9d4ca11032917bbe1543513b0793d8895af9374c1d92ff not found: ID does not exist"
Nov 03 22:37:16.281289 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:16.281258    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7"]
Nov 03 22:37:16.281620 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:16.281603    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-rltc7"]
Nov 03 22:37:17.304598 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:17.304558    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:37:17.934032 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:17.934006    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-2d6076-d576997dd-rh48p"
Nov 03 22:37:18.096651 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:18.096615    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6634b1a8-4b77-4281-8566-3470ecf3022f" path="/var/lib/kubelet/pods/6634b1a8-4b77-4281-8566-3470ecf3022f/volumes"
Nov 03 22:37:18.820770 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:18.820733    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:37:19.022148 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:19.022119    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8b1efeb6-98884dc8c-v2chc"
Nov 03 22:37:20.398514 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:20.398458    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:37:21.850784 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:21.850738    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:37:23.439984 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:23.439946    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:37:25.099278 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:25.095602    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:37:26.756963 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:26.756922    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:37:28.110599 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:28.110564    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:37:28.638500 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:28.638462    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:37:28.781813 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:28.781778    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:37:28.939481 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:28.939447    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:37:29.251805 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:29.251660    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:37:30.604957 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:30.604912    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:37:31.261134 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:37:31.261099    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:37:32.185099 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:32.185043    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:37:33.794147 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:33.794108    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:37:35.242694 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:35.242654    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:37:36.022230 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:36.022182    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:37:36.158847 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:36.158815    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:37:36.328389 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:36.328289    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb"]
Nov 03 22:37:36.328868 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:36.328625    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb" podUID="f44ff83c-d054-4f20-a04c-c6204de2bae2" containerName="istio-proxy" containerID="cri-o://42a5da8b666c75ce24ff12a58e032fdc19ad1897c395692e3852b6f7cb45284e" gracePeriod=30
Nov 03 22:37:36.377914 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:36.377877    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:37:36.955077 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:36.955041    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:37:38.796579 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:38.796541    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:37:40.458934 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:40.458733    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:37:41.621695 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:41.621669    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb"
Nov 03 22:37:41.775998 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:41.775965    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f44ff83c-d054-4f20-a04c-c6204de2bae2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f44ff83c-d054-4f20-a04c-c6204de2bae2-istiod-ca-cert\") pod \"f44ff83c-d054-4f20-a04c-c6204de2bae2\" (UID: \"f44ff83c-d054-4f20-a04c-c6204de2bae2\") "
Nov 03 22:37:41.776183 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:41.776018    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f44ff83c-d054-4f20-a04c-c6204de2bae2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f44ff83c-d054-4f20-a04c-c6204de2bae2-workload-socket\") pod \"f44ff83c-d054-4f20-a04c-c6204de2bae2\" (UID: \"f44ff83c-d054-4f20-a04c-c6204de2bae2\") "
Nov 03 22:37:41.776341 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:41.776318    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f44ff83c-d054-4f20-a04c-c6204de2bae2-istiod-ca-cert" pod "f44ff83c-d054-4f20-a04c-c6204de2bae2" (UID: "f44ff83c-d054-4f20-a04c-c6204de2bae2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:37:41.776399 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:41.776363    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f44ff83c-d054-4f20-a04c-c6204de2bae2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f44ff83c-d054-4f20-a04c-c6204de2bae2-istio-envoy\") pod \"f44ff83c-d054-4f20-a04c-c6204de2bae2\" (UID: \"f44ff83c-d054-4f20-a04c-c6204de2bae2\") "
Nov 03 22:37:41.776399 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:41.776381    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f44ff83c-d054-4f20-a04c-c6204de2bae2-workload-socket" pod "f44ff83c-d054-4f20-a04c-c6204de2bae2" (UID: "f44ff83c-d054-4f20-a04c-c6204de2bae2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:37:41.776483 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:41.776398    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f44ff83c-d054-4f20-a04c-c6204de2bae2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f44ff83c-d054-4f20-a04c-c6204de2bae2-credential-socket\") pod \"f44ff83c-d054-4f20-a04c-c6204de2bae2\" (UID: \"f44ff83c-d054-4f20-a04c-c6204de2bae2\") "
Nov 03 22:37:41.776520 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:41.776504    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f44ff83c-d054-4f20-a04c-c6204de2bae2-kube-api-access-kcf58\" (UniqueName: \"kubernetes.io/projected/f44ff83c-d054-4f20-a04c-c6204de2bae2-kube-api-access-kcf58\") pod \"f44ff83c-d054-4f20-a04c-c6204de2bae2\" (UID: \"f44ff83c-d054-4f20-a04c-c6204de2bae2\") "
Nov 03 22:37:41.776582 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:41.776566    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f44ff83c-d054-4f20-a04c-c6204de2bae2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f44ff83c-d054-4f20-a04c-c6204de2bae2-istio-podinfo\") pod \"f44ff83c-d054-4f20-a04c-c6204de2bae2\" (UID: \"f44ff83c-d054-4f20-a04c-c6204de2bae2\") "
Nov 03 22:37:41.776636 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:41.776594    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f44ff83c-d054-4f20-a04c-c6204de2bae2-credential-socket" pod "f44ff83c-d054-4f20-a04c-c6204de2bae2" (UID: "f44ff83c-d054-4f20-a04c-c6204de2bae2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:37:41.776636 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:41.776609    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f44ff83c-d054-4f20-a04c-c6204de2bae2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f44ff83c-d054-4f20-a04c-c6204de2bae2-workload-certs\") pod \"f44ff83c-d054-4f20-a04c-c6204de2bae2\" (UID: \"f44ff83c-d054-4f20-a04c-c6204de2bae2\") "
Nov 03 22:37:41.776740 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:41.776643    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f44ff83c-d054-4f20-a04c-c6204de2bae2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f44ff83c-d054-4f20-a04c-c6204de2bae2-istio-data\") pod \"f44ff83c-d054-4f20-a04c-c6204de2bae2\" (UID: \"f44ff83c-d054-4f20-a04c-c6204de2bae2\") "
Nov 03 22:37:41.776740 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:41.776695    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f44ff83c-d054-4f20-a04c-c6204de2bae2-istio-token\" (UniqueName: \"kubernetes.io/projected/f44ff83c-d054-4f20-a04c-c6204de2bae2-istio-token\") pod \"f44ff83c-d054-4f20-a04c-c6204de2bae2\" (UID: \"f44ff83c-d054-4f20-a04c-c6204de2bae2\") "
Nov 03 22:37:41.776874 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:41.776847    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f44ff83c-d054-4f20-a04c-c6204de2bae2-workload-certs" pod "f44ff83c-d054-4f20-a04c-c6204de2bae2" (UID: "f44ff83c-d054-4f20-a04c-c6204de2bae2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:37:41.776959 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:41.776940    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f44ff83c-d054-4f20-a04c-c6204de2bae2-istio-data" pod "f44ff83c-d054-4f20-a04c-c6204de2bae2" (UID: "f44ff83c-d054-4f20-a04c-c6204de2bae2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:37:41.777399 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:41.777378    4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f44ff83c-d054-4f20-a04c-c6204de2bae2-credential-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:37:41.777542 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:41.777528    4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f44ff83c-d054-4f20-a04c-c6204de2bae2-workload-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:37:41.777629 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:41.777546    4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f44ff83c-d054-4f20-a04c-c6204de2bae2-istio-data\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:37:41.777629 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:41.777558    4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f44ff83c-d054-4f20-a04c-c6204de2bae2-istiod-ca-cert\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:37:41.777629 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:41.777571    4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f44ff83c-d054-4f20-a04c-c6204de2bae2-workload-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:37:41.778756 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:41.778729    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f44ff83c-d054-4f20-a04c-c6204de2bae2-istio-podinfo" pod "f44ff83c-d054-4f20-a04c-c6204de2bae2" (UID: "f44ff83c-d054-4f20-a04c-c6204de2bae2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 22:37:41.778888 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:41.778784    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f44ff83c-d054-4f20-a04c-c6204de2bae2-kube-api-access-kcf58" pod "f44ff83c-d054-4f20-a04c-c6204de2bae2" (UID: "f44ff83c-d054-4f20-a04c-c6204de2bae2"). InnerVolumeSpecName "kube-api-access-kcf58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:37:41.778951 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:41.778900    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f44ff83c-d054-4f20-a04c-c6204de2bae2-istio-envoy" pod "f44ff83c-d054-4f20-a04c-c6204de2bae2" (UID: "f44ff83c-d054-4f20-a04c-c6204de2bae2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:37:41.779027 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:41.779010    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f44ff83c-d054-4f20-a04c-c6204de2bae2-istio-token" pod "f44ff83c-d054-4f20-a04c-c6204de2bae2" (UID: "f44ff83c-d054-4f20-a04c-c6204de2bae2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:37:41.879015 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:41.878923    4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f44ff83c-d054-4f20-a04c-c6204de2bae2-istio-envoy\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:37:41.879015 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:41.878969    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcf58\" (UniqueName: \"kubernetes.io/projected/f44ff83c-d054-4f20-a04c-c6204de2bae2-kube-api-access-kcf58\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:37:41.879015 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:41.878981    4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f44ff83c-d054-4f20-a04c-c6204de2bae2-istio-podinfo\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:37:41.879015 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:41.878990    4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f44ff83c-d054-4f20-a04c-c6204de2bae2-istio-token\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:37:42.011943 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:42.011906    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:37:42.287535 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:42.287500    4431 generic.go:356] "Generic (PLEG): container finished" podID="f44ff83c-d054-4f20-a04c-c6204de2bae2" containerID="42a5da8b666c75ce24ff12a58e032fdc19ad1897c395692e3852b6f7cb45284e" exitCode=0
Nov 03 22:37:42.287696 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:42.287553    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb"
Nov 03 22:37:42.287696 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:42.287579    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb" event={"ID":"f44ff83c-d054-4f20-a04c-c6204de2bae2","Type":"ContainerDied","Data":"42a5da8b666c75ce24ff12a58e032fdc19ad1897c395692e3852b6f7cb45284e"}
Nov 03 22:37:42.287696 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:42.287633    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb" event={"ID":"f44ff83c-d054-4f20-a04c-c6204de2bae2","Type":"ContainerDied","Data":"baf73975af674c319d2f697566bae2366c90d133227035695897b85952f23e97"}
Nov 03 22:37:42.287696 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:42.287654    4431 scope.go:122] "RemoveContainer" containerID="42a5da8b666c75ce24ff12a58e032fdc19ad1897c395692e3852b6f7cb45284e"
Nov 03 22:37:42.298263 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:42.298210    4431 scope.go:122] "RemoveContainer" containerID="fd8992222dab0b30bb72530b1349c352047bd2118d3f30adca4cb7b736c3d0cb"
Nov 03 22:37:42.307825 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:42.307808    4431 scope.go:122] "RemoveContainer" containerID="42a5da8b666c75ce24ff12a58e032fdc19ad1897c395692e3852b6f7cb45284e"
Nov 03 22:37:42.308101 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:37:42.308078    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42a5da8b666c75ce24ff12a58e032fdc19ad1897c395692e3852b6f7cb45284e\": container with ID starting with 42a5da8b666c75ce24ff12a58e032fdc19ad1897c395692e3852b6f7cb45284e not found: ID does not exist" containerID="42a5da8b666c75ce24ff12a58e032fdc19ad1897c395692e3852b6f7cb45284e"
Nov 03 22:37:42.308167 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:42.308111    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42a5da8b666c75ce24ff12a58e032fdc19ad1897c395692e3852b6f7cb45284e"} err="failed to get container status \"42a5da8b666c75ce24ff12a58e032fdc19ad1897c395692e3852b6f7cb45284e\": rpc error: code = NotFound desc = could not find container \"42a5da8b666c75ce24ff12a58e032fdc19ad1897c395692e3852b6f7cb45284e\": container with ID starting with 42a5da8b666c75ce24ff12a58e032fdc19ad1897c395692e3852b6f7cb45284e not found: ID does not exist"
Nov 03 22:37:42.308167 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:42.308127    4431 scope.go:122] "RemoveContainer" containerID="fd8992222dab0b30bb72530b1349c352047bd2118d3f30adca4cb7b736c3d0cb"
Nov 03 22:37:42.308400 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:37:42.308381    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd8992222dab0b30bb72530b1349c352047bd2118d3f30adca4cb7b736c3d0cb\": container with ID starting with fd8992222dab0b30bb72530b1349c352047bd2118d3f30adca4cb7b736c3d0cb not found: ID does not exist" containerID="fd8992222dab0b30bb72530b1349c352047bd2118d3f30adca4cb7b736c3d0cb"
Nov 03 22:37:42.308467 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:42.308410    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd8992222dab0b30bb72530b1349c352047bd2118d3f30adca4cb7b736c3d0cb"} err="failed to get container status \"fd8992222dab0b30bb72530b1349c352047bd2118d3f30adca4cb7b736c3d0cb\": rpc error: code = NotFound desc = could not find container \"fd8992222dab0b30bb72530b1349c352047bd2118d3f30adca4cb7b736c3d0cb\": container with ID starting with fd8992222dab0b30bb72530b1349c352047bd2118d3f30adca4cb7b736c3d0cb not found: ID does not exist"
Nov 03 22:37:42.342039 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:42.342011    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb"]
Nov 03 22:37:42.354317 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:42.354290    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-4c5345-openshift-default-86899fff89-l8tzb"]
Nov 03 22:37:42.548473 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:42.548385    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:37:43.635112 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:43.635043    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:37:44.088858 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:44.088822    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f44ff83c-d054-4f20-a04c-c6204de2bae2" path="/var/lib/kubelet/pods/f44ff83c-d054-4f20-a04c-c6204de2bae2/volumes"
Nov 03 22:37:45.188804 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:45.188768    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2d6076-5497f8bb66-mbzbx"]
Nov 03 22:37:45.189500 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:45.189484    4431 cpu_manager.go:414] "containerMap: removing container" podUID="f44ff83c-d054-4f20-a04c-c6204de2bae2" containerName="istio-proxy"
Nov 03 22:37:45.189500 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:45.189501    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f44ff83c-d054-4f20-a04c-c6204de2bae2" containerName="istio-proxy"
Nov 03 22:37:45.189604 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:45.189544    4431 cpu_manager.go:414] "containerMap: removing container" podUID="6634b1a8-4b77-4281-8566-3470ecf3022f" containerName="istio-proxy"
Nov 03 22:37:45.189604 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:45.189549    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6634b1a8-4b77-4281-8566-3470ecf3022f" containerName="istio-proxy"
Nov 03 22:37:45.189678 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:45.189669    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="6634b1a8-4b77-4281-8566-3470ecf3022f" containerName="istio-proxy"
Nov 03 22:37:45.189718 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:45.189684    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="f44ff83c-d054-4f20-a04c-c6204de2bae2" containerName="istio-proxy"
Nov 03 22:37:45.193922 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:45.193893    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2d6076-5497f8bb66-mbzbx"
Nov 03 22:37:45.211152 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:45.211131    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-2d6076-serving-cert\""
Nov 03 22:37:45.225271 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:45.225241    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2d6076-5497f8bb66-mbzbx"]
Nov 03 22:37:45.323679 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:45.323647    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:37:45.331092 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:45.331068    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23ec5be6-eefc-425a-a820-f639d9a0bcf3-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2d6076-5497f8bb66-mbzbx\" (UID: \"23ec5be6-eefc-425a-a820-f639d9a0bcf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2d6076-5497f8bb66-mbzbx"
Nov 03 22:37:45.331241 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:45.331126    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23ec5be6-eefc-425a-a820-f639d9a0bcf3-metrics-certs\") pod \"maas-api-e2e-inf-b-2d6076-5497f8bb66-mbzbx\" (UID: \"23ec5be6-eefc-425a-a820-f639d9a0bcf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2d6076-5497f8bb66-mbzbx"
Nov 03 22:37:45.331302 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:45.331280    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmcfk\" (UniqueName: \"kubernetes.io/projected/23ec5be6-eefc-425a-a820-f639d9a0bcf3-kube-api-access-jmcfk\") pod \"maas-api-e2e-inf-b-2d6076-5497f8bb66-mbzbx\" (UID: \"23ec5be6-eefc-425a-a820-f639d9a0bcf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2d6076-5497f8bb66-mbzbx"
Nov 03 22:37:45.331449 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:45.331431    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23ec5be6-eefc-425a-a820-f639d9a0bcf3-maas-api-tls\") pod \"maas-api-e2e-inf-b-2d6076-5497f8bb66-mbzbx\" (UID: \"23ec5be6-eefc-425a-a820-f639d9a0bcf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2d6076-5497f8bb66-mbzbx"
Nov 03 22:37:45.433194 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:45.433159    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23ec5be6-eefc-425a-a820-f639d9a0bcf3-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2d6076-5497f8bb66-mbzbx\" (UID: \"23ec5be6-eefc-425a-a820-f639d9a0bcf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2d6076-5497f8bb66-mbzbx"
Nov 03 22:37:45.433403 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:45.433245    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23ec5be6-eefc-425a-a820-f639d9a0bcf3-metrics-certs\") pod \"maas-api-e2e-inf-b-2d6076-5497f8bb66-mbzbx\" (UID: \"23ec5be6-eefc-425a-a820-f639d9a0bcf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2d6076-5497f8bb66-mbzbx"
Nov 03 22:37:45.433403 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:45.433311    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmcfk\" (UniqueName: \"kubernetes.io/projected/23ec5be6-eefc-425a-a820-f639d9a0bcf3-kube-api-access-jmcfk\") pod \"maas-api-e2e-inf-b-2d6076-5497f8bb66-mbzbx\" (UID: \"23ec5be6-eefc-425a-a820-f639d9a0bcf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2d6076-5497f8bb66-mbzbx"
Nov 03 22:37:45.433526 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:45.433435    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23ec5be6-eefc-425a-a820-f639d9a0bcf3-maas-api-tls\") pod \"maas-api-e2e-inf-b-2d6076-5497f8bb66-mbzbx\" (UID: \"23ec5be6-eefc-425a-a820-f639d9a0bcf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2d6076-5497f8bb66-mbzbx"
Nov 03 22:37:45.434074 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:45.434037    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23ec5be6-eefc-425a-a820-f639d9a0bcf3-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2d6076-5497f8bb66-mbzbx\" (UID: \"23ec5be6-eefc-425a-a820-f639d9a0bcf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2d6076-5497f8bb66-mbzbx"
Nov 03 22:37:45.436068 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:45.436025    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23ec5be6-eefc-425a-a820-f639d9a0bcf3-metrics-certs\") pod \"maas-api-e2e-inf-b-2d6076-5497f8bb66-mbzbx\" (UID: \"23ec5be6-eefc-425a-a820-f639d9a0bcf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2d6076-5497f8bb66-mbzbx"
Nov 03 22:37:45.436555 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:45.436529    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23ec5be6-eefc-425a-a820-f639d9a0bcf3-maas-api-tls\") pod \"maas-api-e2e-inf-b-2d6076-5497f8bb66-mbzbx\" (UID: \"23ec5be6-eefc-425a-a820-f639d9a0bcf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2d6076-5497f8bb66-mbzbx"
Nov 03 22:37:45.456569 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:45.456504    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmcfk\" (UniqueName: \"kubernetes.io/projected/23ec5be6-eefc-425a-a820-f639d9a0bcf3-kube-api-access-jmcfk\") pod \"maas-api-e2e-inf-b-2d6076-5497f8bb66-mbzbx\" (UID: \"23ec5be6-eefc-425a-a820-f639d9a0bcf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2d6076-5497f8bb66-mbzbx"
Nov 03 22:37:45.469461 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:45.469434    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:37:45.508020 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:45.507992    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2d6076-5497f8bb66-mbzbx"
Nov 03 22:37:45.627036 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:45.626260    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:37:45.858023 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:45.857989    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2d6076-5497f8bb66-mbzbx"]
Nov 03 22:37:45.858483 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:37:45.858448    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23ec5be6_eefc_425a_a820_f639d9a0bcf3.slice/crio-72bfc86c9972e9cc5b795169623e19be11e87346d5134920d7a1450a5fc45315 WatchSource:0}: Error finding container 72bfc86c9972e9cc5b795169623e19be11e87346d5134920d7a1450a5fc45315: Status 404 returned error can't find the container with id 72bfc86c9972e9cc5b795169623e19be11e87346d5134920d7a1450a5fc45315
Nov 03 22:37:46.318504 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:46.318462    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2d6076-5497f8bb66-mbzbx" event={"ID":"23ec5be6-eefc-425a-a820-f639d9a0bcf3","Type":"ContainerStarted","Data":"8f3e8703bd1f2e18a8ada8eb055c5b0a9a3e723e9927b89b5406c2f66ae25cb0"}
Nov 03 22:37:46.318893 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:46.318528    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2d6076-5497f8bb66-mbzbx" event={"ID":"23ec5be6-eefc-425a-a820-f639d9a0bcf3","Type":"ContainerStarted","Data":"72bfc86c9972e9cc5b795169623e19be11e87346d5134920d7a1450a5fc45315"}
Nov 03 22:37:46.318893 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:46.318562    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2d6076-5497f8bb66-mbzbx"
Nov 03 22:37:46.384183 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:46.384115    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2d6076-5497f8bb66-mbzbx" podStartSLOduration=1.093158145 podStartE2EDuration="1.384093405s" totalImagesPullingTime="290.93526ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:37:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:37:46.373976553 +0000 UTC m=+1490.399832418" watchObservedRunningTime="2026-11-03 22:37:46.384093405 +0000 UTC m=+1490.409949270"
Nov 03 22:37:47.060464 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:47.060419    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:37:47.237883 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:47.235627    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:37:48.028891 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:48.028852    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bbfc2e-6ddb5f6584-hc4kx"]
Nov 03 22:37:48.192951 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:48.192913    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:37:48.337545 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:48.337436    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:37:48.840241 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:48.833674    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:37:50.421203 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:50.421165    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:37:51.799164 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:51.799132    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:37:52.076109 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:52.076020    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:37:52.330984 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:52.330902    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2d6076-5497f8bb66-mbzbx"
Nov 03 22:37:53.029748 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:53.029705    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-bbfc2e-6ddb5f6584-hc4kx" podUID="7e7cb728-8ff5-431d-80d8-87cffee7f726" containerName="payload-processing" containerID="cri-o://14a52d98b83639dedc5ef62330b48a9c22cac48c01b8886ef0b8b8657d4b9121" gracePeriod=25
Nov 03 22:37:53.166144 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:53.166115    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-22edc1-684d7dd455-c6bwb"]
Nov 03 22:37:53.167147 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:53.167122    4431 cpu_manager.go:414] "containerMap: removing container" podUID="f44ff83c-d054-4f20-a04c-c6204de2bae2" containerName="istio-proxy"
Nov 03 22:37:53.167147 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:53.167144    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f44ff83c-d054-4f20-a04c-c6204de2bae2" containerName="istio-proxy"
Nov 03 22:37:53.167363 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:53.167278    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="f44ff83c-d054-4f20-a04c-c6204de2bae2" containerName="istio-proxy"
Nov 03 22:37:53.170879 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:53.170862    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22edc1-684d7dd455-c6bwb"
Nov 03 22:37:53.193965 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:53.193928    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-22edc1-684d7dd455-c6bwb"]
Nov 03 22:37:53.200797 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:53.199688    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-22edc1-serving-cert\""
Nov 03 22:37:53.268123 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:53.268086    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhbr7\" (UniqueName: \"kubernetes.io/projected/a1475fef-4430-4907-9ddc-41aba6c3562c-kube-api-access-mhbr7\") pod \"maas-api-e2e-api-b-22edc1-684d7dd455-c6bwb\" (UID: \"a1475fef-4430-4907-9ddc-41aba6c3562c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22edc1-684d7dd455-c6bwb"
Nov 03 22:37:53.268369 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:53.268252    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1475fef-4430-4907-9ddc-41aba6c3562c-metrics-certs\") pod \"maas-api-e2e-api-b-22edc1-684d7dd455-c6bwb\" (UID: \"a1475fef-4430-4907-9ddc-41aba6c3562c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22edc1-684d7dd455-c6bwb"
Nov 03 22:37:53.268369 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:53.268347    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1475fef-4430-4907-9ddc-41aba6c3562c-openshift-service-ca\") pod \"maas-api-e2e-api-b-22edc1-684d7dd455-c6bwb\" (UID: \"a1475fef-4430-4907-9ddc-41aba6c3562c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22edc1-684d7dd455-c6bwb"
Nov 03 22:37:53.268511 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:53.268489    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1475fef-4430-4907-9ddc-41aba6c3562c-maas-api-tls\") pod \"maas-api-e2e-api-b-22edc1-684d7dd455-c6bwb\" (UID: \"a1475fef-4430-4907-9ddc-41aba6c3562c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22edc1-684d7dd455-c6bwb"
Nov 03 22:37:53.280081 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:53.279997    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:37:53.350743 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:53.350714    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-bbfc2e-6ddb5f6584-hc4kx"
Nov 03 22:37:53.374242 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:53.374052    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e7cb728-8ff5-431d-80d8-87cffee7f726-kube-api-access-crm7c\" (UniqueName: \"kubernetes.io/projected/7e7cb728-8ff5-431d-80d8-87cffee7f726-kube-api-access-crm7c\") pod \"7e7cb728-8ff5-431d-80d8-87cffee7f726\" (UID: \"7e7cb728-8ff5-431d-80d8-87cffee7f726\") "
Nov 03 22:37:53.374448 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:53.374251    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e7cb728-8ff5-431d-80d8-87cffee7f726-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e7cb728-8ff5-431d-80d8-87cffee7f726-plugins-config-volume\") pod \"7e7cb728-8ff5-431d-80d8-87cffee7f726\" (UID: \"7e7cb728-8ff5-431d-80d8-87cffee7f726\") "
Nov 03 22:37:53.385239 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:53.374712    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1475fef-4430-4907-9ddc-41aba6c3562c-maas-api-tls\") pod \"maas-api-e2e-api-b-22edc1-684d7dd455-c6bwb\" (UID: \"a1475fef-4430-4907-9ddc-41aba6c3562c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22edc1-684d7dd455-c6bwb"
Nov 03 22:37:53.385239 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:53.374969    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhbr7\" (UniqueName: \"kubernetes.io/projected/a1475fef-4430-4907-9ddc-41aba6c3562c-kube-api-access-mhbr7\") pod \"maas-api-e2e-api-b-22edc1-684d7dd455-c6bwb\" (UID: \"a1475fef-4430-4907-9ddc-41aba6c3562c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22edc1-684d7dd455-c6bwb"
Nov 03 22:37:53.385239 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:53.375134    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1475fef-4430-4907-9ddc-41aba6c3562c-metrics-certs\") pod \"maas-api-e2e-api-b-22edc1-684d7dd455-c6bwb\" (UID: \"a1475fef-4430-4907-9ddc-41aba6c3562c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22edc1-684d7dd455-c6bwb"
Nov 03 22:37:53.385239 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:53.375171    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1475fef-4430-4907-9ddc-41aba6c3562c-openshift-service-ca\") pod \"maas-api-e2e-api-b-22edc1-684d7dd455-c6bwb\" (UID: \"a1475fef-4430-4907-9ddc-41aba6c3562c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22edc1-684d7dd455-c6bwb"
Nov 03 22:37:53.385239 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:53.376170    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1475fef-4430-4907-9ddc-41aba6c3562c-openshift-service-ca\") pod \"maas-api-e2e-api-b-22edc1-684d7dd455-c6bwb\" (UID: \"a1475fef-4430-4907-9ddc-41aba6c3562c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22edc1-684d7dd455-c6bwb"
Nov 03 22:37:53.388480 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:53.386543    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1475fef-4430-4907-9ddc-41aba6c3562c-metrics-certs\") pod \"maas-api-e2e-api-b-22edc1-684d7dd455-c6bwb\" (UID: \"a1475fef-4430-4907-9ddc-41aba6c3562c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22edc1-684d7dd455-c6bwb"
Nov 03 22:37:53.401304 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:53.396445    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1475fef-4430-4907-9ddc-41aba6c3562c-maas-api-tls\") pod \"maas-api-e2e-api-b-22edc1-684d7dd455-c6bwb\" (UID: \"a1475fef-4430-4907-9ddc-41aba6c3562c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22edc1-684d7dd455-c6bwb"
Nov 03 22:37:53.401304 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:53.399069    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e7cb728-8ff5-431d-80d8-87cffee7f726-plugins-config-volume" pod "7e7cb728-8ff5-431d-80d8-87cffee7f726" (UID: "7e7cb728-8ff5-431d-80d8-87cffee7f726"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:37:53.408067 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:53.403652    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e7cb728-8ff5-431d-80d8-87cffee7f726-kube-api-access-crm7c" pod "7e7cb728-8ff5-431d-80d8-87cffee7f726" (UID: "7e7cb728-8ff5-431d-80d8-87cffee7f726"). InnerVolumeSpecName "kube-api-access-crm7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:37:53.408067 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:53.407679    4431 generic.go:356] "Generic (PLEG): container finished" podID="7e7cb728-8ff5-431d-80d8-87cffee7f726" containerID="14a52d98b83639dedc5ef62330b48a9c22cac48c01b8886ef0b8b8657d4b9121" exitCode=0
Nov 03 22:37:53.408067 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:53.407749    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-bbfc2e-6ddb5f6584-hc4kx"
Nov 03 22:37:53.408067 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:53.407792    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-bbfc2e-6ddb5f6584-hc4kx" event={"ID":"7e7cb728-8ff5-431d-80d8-87cffee7f726","Type":"ContainerDied","Data":"14a52d98b83639dedc5ef62330b48a9c22cac48c01b8886ef0b8b8657d4b9121"}
Nov 03 22:37:53.408067 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:53.407867    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-bbfc2e-6ddb5f6584-hc4kx" event={"ID":"7e7cb728-8ff5-431d-80d8-87cffee7f726","Type":"ContainerDied","Data":"68d3e20553b9724caa8daf15a37e284a7f7d73924a3c45ce7faae645bdb238b9"}
Nov 03 22:37:53.408067 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:53.407898    4431 scope.go:122] "RemoveContainer" containerID="14a52d98b83639dedc5ef62330b48a9c22cac48c01b8886ef0b8b8657d4b9121"
Nov 03 22:37:53.458600 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:53.458560    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:37:53.458776 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:53.458616    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhbr7\" (UniqueName: \"kubernetes.io/projected/a1475fef-4430-4907-9ddc-41aba6c3562c-kube-api-access-mhbr7\") pod \"maas-api-e2e-api-b-22edc1-684d7dd455-c6bwb\" (UID: \"a1475fef-4430-4907-9ddc-41aba6c3562c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22edc1-684d7dd455-c6bwb"
Nov 03 22:37:53.474642 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:53.474615    4431 scope.go:122] "RemoveContainer" containerID="14a52d98b83639dedc5ef62330b48a9c22cac48c01b8886ef0b8b8657d4b9121"
Nov 03 22:37:53.475014 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:37:53.474986    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14a52d98b83639dedc5ef62330b48a9c22cac48c01b8886ef0b8b8657d4b9121\": container with ID starting with 14a52d98b83639dedc5ef62330b48a9c22cac48c01b8886ef0b8b8657d4b9121 not found: ID does not exist" containerID="14a52d98b83639dedc5ef62330b48a9c22cac48c01b8886ef0b8b8657d4b9121"
Nov 03 22:37:53.475101 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:53.475026    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14a52d98b83639dedc5ef62330b48a9c22cac48c01b8886ef0b8b8657d4b9121"} err="failed to get container status \"14a52d98b83639dedc5ef62330b48a9c22cac48c01b8886ef0b8b8657d4b9121\": rpc error: code = NotFound desc = could not find container \"14a52d98b83639dedc5ef62330b48a9c22cac48c01b8886ef0b8b8657d4b9121\": container with ID starting with 14a52d98b83639dedc5ef62330b48a9c22cac48c01b8886ef0b8b8657d4b9121 not found: ID does not exist"
Nov 03 22:37:53.477033 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:53.477008    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crm7c\" (UniqueName: \"kubernetes.io/projected/7e7cb728-8ff5-431d-80d8-87cffee7f726-kube-api-access-crm7c\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:37:53.477033 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:53.477036    4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e7cb728-8ff5-431d-80d8-87cffee7f726-plugins-config-volume\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:37:53.487615 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:53.487586    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22edc1-684d7dd455-c6bwb"
Nov 03 22:37:53.503988 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:53.503953    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bbfc2e-6ddb5f6584-hc4kx"]
Nov 03 22:37:53.510963 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:53.510924    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bbfc2e-6ddb5f6584-hc4kx"]
Nov 03 22:37:53.812652 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:53.812590    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-22edc1-684d7dd455-c6bwb"]
Nov 03 22:37:53.812785 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:37:53.812719    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1475fef_4430_4907_9ddc_41aba6c3562c.slice/crio-46131b0ae91a4389beadca32c01f896b84e7f65d26a9f376241c43b5b3d5d9e2 WatchSource:0}: Error finding container 46131b0ae91a4389beadca32c01f896b84e7f65d26a9f376241c43b5b3d5d9e2: Status 404 returned error can't find the container with id 46131b0ae91a4389beadca32c01f896b84e7f65d26a9f376241c43b5b3d5d9e2
Nov 03 22:37:54.095333 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:54.095247    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e7cb728-8ff5-431d-80d8-87cffee7f726" path="/var/lib/kubelet/pods/7e7cb728-8ff5-431d-80d8-87cffee7f726/volumes"
Nov 03 22:37:54.425754 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:54.425713    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22edc1-684d7dd455-c6bwb" event={"ID":"a1475fef-4430-4907-9ddc-41aba6c3562c","Type":"ContainerStarted","Data":"46131b0ae91a4389beadca32c01f896b84e7f65d26a9f376241c43b5b3d5d9e2"}
Nov 03 22:37:54.425926 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:54.425826    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22edc1-684d7dd455-c6bwb"
Nov 03 22:37:54.473047 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:54.472985    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22edc1-684d7dd455-c6bwb" podStartSLOduration=1.0521585980000001 podStartE2EDuration="1.472970467s" totalImagesPullingTime="420.811869ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:37:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:37:54.467631093 +0000 UTC m=+1498.493486956" watchObservedRunningTime="2026-11-03 22:37:54.472970467 +0000 UTC m=+1498.498826328"
Nov 03 22:37:54.823709 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:54.823668    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:37:54.967976 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:54.967940    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:37:55.452238 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:55.450759    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22edc1-684d7dd455-c6bwb" event={"ID":"a1475fef-4430-4907-9ddc-41aba6c3562c","Type":"ContainerStarted","Data":"a08f6ead02b41cf57ffe2c23c56816520f574187b52d3a21064b6fe40657877c"}
Nov 03 22:37:56.537374 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:56.537330    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:37:56.588528 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:56.584288    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mx5q8_ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea/ovn-acl-logging/0.log"
Nov 03 22:37:56.593231 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:56.593202    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mx5q8_ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea/ovn-acl-logging/0.log"
Nov 03 22:37:58.668387 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:37:58.668353    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:38:00.517935 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:00.517892    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:38:00.668573 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:00.668534    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:38:01.262643 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:38:01.262596    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:38:01.515823 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:01.515749    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22edc1-684d7dd455-c6bwb"
Nov 03 22:38:02.170128 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:02.170071    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:38:03.817562 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:03.817522    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:38:05.498164 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:05.498103    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:38:07.274274 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:07.274199    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:38:09.052402 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:09.049587    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:38:10.693555 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:10.693513    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:38:12.177235 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:12.177187    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:38:13.298619 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:13.298585    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-65b0e2-765dd6d498-5n54k"]
Nov 03 22:38:13.338592 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:13.338560    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-65b0e2-5fc849b4cf-m2md4"]
Nov 03 22:38:13.465368 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:13.465324    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:38:13.606709 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:13.606626    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:38:13.747186 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:13.747151    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:38:15.371375 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:15.371339    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:38:16.973406 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:16.973370    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:38:17.703509 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:17.703471    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:38:17.846452 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:17.846070    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:38:18.300300 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:18.299162    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-65b0e2-765dd6d498-5n54k" podUID="22f9e7cf-d329-4178-af61-b9c27d485907" containerName="payload-processing" containerID="cri-o://9970259817a937954e15c1b8b5a919a3c6ff32ee87efdb304cbaf9d12347f7f3" gracePeriod=25
Nov 03 22:38:18.341555 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:18.339452    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-65b0e2-5fc849b4cf-m2md4" podUID="c907cfe8-b91c-4539-8503-20eb6561f3ad" containerName="payload-pre-processing" containerID="cri-o://44f147fc2e0e54f09b61b3a27bee7e6ba7f0fa224cb5a2e65b1a05fb31fa7cc2" gracePeriod=25
Nov 03 22:38:18.644201 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:18.643957    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:38:18.702745 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:18.702707    4431 generic.go:356] "Generic (PLEG): container finished" podID="22f9e7cf-d329-4178-af61-b9c27d485907" containerID="9970259817a937954e15c1b8b5a919a3c6ff32ee87efdb304cbaf9d12347f7f3" exitCode=0
Nov 03 22:38:18.702936 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:18.702775    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-65b0e2-765dd6d498-5n54k"
Nov 03 22:38:18.702936 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:18.702853    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-65b0e2-765dd6d498-5n54k" event={"ID":"22f9e7cf-d329-4178-af61-b9c27d485907","Type":"ContainerDied","Data":"9970259817a937954e15c1b8b5a919a3c6ff32ee87efdb304cbaf9d12347f7f3"}
Nov 03 22:38:18.702936 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:18.702907    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-65b0e2-765dd6d498-5n54k" event={"ID":"22f9e7cf-d329-4178-af61-b9c27d485907","Type":"ContainerDied","Data":"38fa57259914628e5ac9794ae78edfe9fc6d18fbde4a5d3a17a0155e1cd06f6c"}
Nov 03 22:38:18.702936 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:18.702932    4431 scope.go:122] "RemoveContainer" containerID="9970259817a937954e15c1b8b5a919a3c6ff32ee87efdb304cbaf9d12347f7f3"
Nov 03 22:38:18.705662 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:18.705629    4431 generic.go:356] "Generic (PLEG): container finished" podID="c907cfe8-b91c-4539-8503-20eb6561f3ad" containerID="44f147fc2e0e54f09b61b3a27bee7e6ba7f0fa224cb5a2e65b1a05fb31fa7cc2" exitCode=0
Nov 03 22:38:18.705878 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:18.705856    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-65b0e2-5fc849b4cf-m2md4" event={"ID":"c907cfe8-b91c-4539-8503-20eb6561f3ad","Type":"ContainerDied","Data":"44f147fc2e0e54f09b61b3a27bee7e6ba7f0fa224cb5a2e65b1a05fb31fa7cc2"}
Nov 03 22:38:18.734402 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:18.734376    4431 scope.go:122] "RemoveContainer" containerID="9970259817a937954e15c1b8b5a919a3c6ff32ee87efdb304cbaf9d12347f7f3"
Nov 03 22:38:18.734854 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:38:18.734818    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9970259817a937954e15c1b8b5a919a3c6ff32ee87efdb304cbaf9d12347f7f3\": container with ID starting with 9970259817a937954e15c1b8b5a919a3c6ff32ee87efdb304cbaf9d12347f7f3 not found: ID does not exist" containerID="9970259817a937954e15c1b8b5a919a3c6ff32ee87efdb304cbaf9d12347f7f3"
Nov 03 22:38:18.734974 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:18.734862    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9970259817a937954e15c1b8b5a919a3c6ff32ee87efdb304cbaf9d12347f7f3"} err="failed to get container status \"9970259817a937954e15c1b8b5a919a3c6ff32ee87efdb304cbaf9d12347f7f3\": rpc error: code = NotFound desc = could not find container \"9970259817a937954e15c1b8b5a919a3c6ff32ee87efdb304cbaf9d12347f7f3\": container with ID starting with 9970259817a937954e15c1b8b5a919a3c6ff32ee87efdb304cbaf9d12347f7f3 not found: ID does not exist"
Nov 03 22:38:18.737086 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:18.737062    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-65b0e2-5fc849b4cf-m2md4"
Nov 03 22:38:18.823294 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:18.823254    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22f9e7cf-d329-4178-af61-b9c27d485907-kube-api-access-l6mxx\" (UniqueName: \"kubernetes.io/projected/22f9e7cf-d329-4178-af61-b9c27d485907-kube-api-access-l6mxx\") pod \"22f9e7cf-d329-4178-af61-b9c27d485907\" (UID: \"22f9e7cf-d329-4178-af61-b9c27d485907\") "
Nov 03 22:38:18.823509 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:18.823477    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22f9e7cf-d329-4178-af61-b9c27d485907-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22f9e7cf-d329-4178-af61-b9c27d485907-plugins-config-volume\") pod \"22f9e7cf-d329-4178-af61-b9c27d485907\" (UID: \"22f9e7cf-d329-4178-af61-b9c27d485907\") "
Nov 03 22:38:18.823643 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:18.823620    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c907cfe8-b91c-4539-8503-20eb6561f3ad-kube-api-access-qgxbs\" (UniqueName: \"kubernetes.io/projected/c907cfe8-b91c-4539-8503-20eb6561f3ad-kube-api-access-qgxbs\") pod \"c907cfe8-b91c-4539-8503-20eb6561f3ad\" (UID: \"c907cfe8-b91c-4539-8503-20eb6561f3ad\") "
Nov 03 22:38:18.823791 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:18.823662    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c907cfe8-b91c-4539-8503-20eb6561f3ad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c907cfe8-b91c-4539-8503-20eb6561f3ad-plugins-config-volume\") pod \"c907cfe8-b91c-4539-8503-20eb6561f3ad\" (UID: \"c907cfe8-b91c-4539-8503-20eb6561f3ad\") "
Nov 03 22:38:18.824067 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:18.823901    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22f9e7cf-d329-4178-af61-b9c27d485907-plugins-config-volume" pod "22f9e7cf-d329-4178-af61-b9c27d485907" (UID: "22f9e7cf-d329-4178-af61-b9c27d485907"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:38:18.824165 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:18.824087    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c907cfe8-b91c-4539-8503-20eb6561f3ad-plugins-config-volume" pod "c907cfe8-b91c-4539-8503-20eb6561f3ad" (UID: "c907cfe8-b91c-4539-8503-20eb6561f3ad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:38:18.824435 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:18.824412    4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22f9e7cf-d329-4178-af61-b9c27d485907-plugins-config-volume\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:38:18.824535 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:18.824436    4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c907cfe8-b91c-4539-8503-20eb6561f3ad-plugins-config-volume\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:38:18.825428 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:18.825408    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22f9e7cf-d329-4178-af61-b9c27d485907-kube-api-access-l6mxx" pod "22f9e7cf-d329-4178-af61-b9c27d485907" (UID: "22f9e7cf-d329-4178-af61-b9c27d485907"). InnerVolumeSpecName "kube-api-access-l6mxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:38:18.825895 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:18.825878    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c907cfe8-b91c-4539-8503-20eb6561f3ad-kube-api-access-qgxbs" pod "c907cfe8-b91c-4539-8503-20eb6561f3ad" (UID: "c907cfe8-b91c-4539-8503-20eb6561f3ad"). InnerVolumeSpecName "kube-api-access-qgxbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:38:18.926144 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:18.926106    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6mxx\" (UniqueName: \"kubernetes.io/projected/22f9e7cf-d329-4178-af61-b9c27d485907-kube-api-access-l6mxx\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:38:18.926144 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:18.926140    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgxbs\" (UniqueName: \"kubernetes.io/projected/c907cfe8-b91c-4539-8503-20eb6561f3ad-kube-api-access-qgxbs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:38:19.713473 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:19.713445    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-65b0e2-765dd6d498-5n54k"
Nov 03 22:38:19.715066 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:19.715045    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-65b0e2-5fc849b4cf-m2md4"
Nov 03 22:38:19.715178 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:19.715095    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-65b0e2-5fc849b4cf-m2md4" event={"ID":"c907cfe8-b91c-4539-8503-20eb6561f3ad","Type":"ContainerDied","Data":"92fb86cd2a5e1e32f51b43e26b58d03110a31ec0d3d1a84615f80c854eaf74f5"}
Nov 03 22:38:19.715178 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:19.715171    4431 scope.go:122] "RemoveContainer" containerID="44f147fc2e0e54f09b61b3a27bee7e6ba7f0fa224cb5a2e65b1a05fb31fa7cc2"
Nov 03 22:38:19.766626 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:19.766597    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-65b0e2-5fc849b4cf-m2md4"]
Nov 03 22:38:19.778471 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:19.778444    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-65b0e2-5fc849b4cf-m2md4"]
Nov 03 22:38:19.802821 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:19.802786    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-65b0e2-765dd6d498-5n54k"]
Nov 03 22:38:19.808710 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:19.808681    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-65b0e2-765dd6d498-5n54k"]
Nov 03 22:38:19.942885 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:19.942855    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:38:20.090239 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:20.090149    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22f9e7cf-d329-4178-af61-b9c27d485907" path="/var/lib/kubelet/pods/22f9e7cf-d329-4178-af61-b9c27d485907/volumes"
Nov 03 22:38:20.090694 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:20.090679    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c907cfe8-b91c-4539-8503-20eb6561f3ad" path="/var/lib/kubelet/pods/c907cfe8-b91c-4539-8503-20eb6561f3ad/volumes"
Nov 03 22:38:20.094097 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:20.094071    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:38:20.249292 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:20.248883    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:38:20.644711 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:20.642210    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:38:20.808272 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:20.807609    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:38:22.104674 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:22.104635    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:38:23.847392 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:23.847352    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:38:24.689158 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:24.689128    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:38:24.840737 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:24.840702    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:38:25.795041 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:25.783458    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:38:26.346410 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:26.346380    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:38:26.512055 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:26.512012    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:38:27.917737 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:27.917694    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:38:29.721432 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:29.721395    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:38:31.174204 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:31.174159    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:38:31.261979 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:38:31.261940    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:38:31.524261 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:31.524158    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:38:32.899261 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:32.899204    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:38:33.284706 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:33.284659    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:38:34.604141 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:34.604104    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8b1efeb6-98884dc8c-v2chc"]
Nov 03 22:38:34.604643 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:34.604389    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8b1efeb6-98884dc8c-v2chc" podUID="058f82b5-15b1-4757-8cc9-fcf85f287ef3" containerName="maas-api" containerID="cri-o://cbf63fc7373cc42648112d74ccec6986216a33a37ce6053a512cd012205ee8bf" gracePeriod=30
Nov 03 22:38:34.713757 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:34.713716    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:38:35.096456 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:35.093662    4431 generic.go:356] "Generic (PLEG): container finished" podID="058f82b5-15b1-4757-8cc9-fcf85f287ef3" containerID="cbf63fc7373cc42648112d74ccec6986216a33a37ce6053a512cd012205ee8bf" exitCode=0
Nov 03 22:38:35.096456 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:35.093786    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8b1efeb6-98884dc8c-v2chc" event={"ID":"058f82b5-15b1-4757-8cc9-fcf85f287ef3","Type":"ContainerDied","Data":"cbf63fc7373cc42648112d74ccec6986216a33a37ce6053a512cd012205ee8bf"}
Nov 03 22:38:35.193478 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:35.193445    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8b1efeb6-98884dc8c-v2chc"
Nov 03 22:38:35.196681 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:35.196651    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:38:35.277872 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:35.277778    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/058f82b5-15b1-4757-8cc9-fcf85f287ef3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/058f82b5-15b1-4757-8cc9-fcf85f287ef3-maas-api-tls\") pod \"058f82b5-15b1-4757-8cc9-fcf85f287ef3\" (UID: \"058f82b5-15b1-4757-8cc9-fcf85f287ef3\") "
Nov 03 22:38:35.278758 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:35.278734    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/058f82b5-15b1-4757-8cc9-fcf85f287ef3-kube-api-access-l854b\" (UniqueName: \"kubernetes.io/projected/058f82b5-15b1-4757-8cc9-fcf85f287ef3-kube-api-access-l854b\") pod \"058f82b5-15b1-4757-8cc9-fcf85f287ef3\" (UID: \"058f82b5-15b1-4757-8cc9-fcf85f287ef3\") "
Nov 03 22:38:35.278970 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:35.278955    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/058f82b5-15b1-4757-8cc9-fcf85f287ef3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/058f82b5-15b1-4757-8cc9-fcf85f287ef3-metrics-certs\") pod \"058f82b5-15b1-4757-8cc9-fcf85f287ef3\" (UID: \"058f82b5-15b1-4757-8cc9-fcf85f287ef3\") "
Nov 03 22:38:35.279156 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:35.279143    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/058f82b5-15b1-4757-8cc9-fcf85f287ef3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/058f82b5-15b1-4757-8cc9-fcf85f287ef3-openshift-service-ca\") pod \"058f82b5-15b1-4757-8cc9-fcf85f287ef3\" (UID: \"058f82b5-15b1-4757-8cc9-fcf85f287ef3\") "
Nov 03 22:38:35.280295 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:35.280272    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/058f82b5-15b1-4757-8cc9-fcf85f287ef3-openshift-service-ca" pod "058f82b5-15b1-4757-8cc9-fcf85f287ef3" (UID: "058f82b5-15b1-4757-8cc9-fcf85f287ef3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:38:35.285580 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:35.285554    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/058f82b5-15b1-4757-8cc9-fcf85f287ef3-metrics-certs" pod "058f82b5-15b1-4757-8cc9-fcf85f287ef3" (UID: "058f82b5-15b1-4757-8cc9-fcf85f287ef3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:38:35.289782 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:35.289759    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/058f82b5-15b1-4757-8cc9-fcf85f287ef3-kube-api-access-l854b" pod "058f82b5-15b1-4757-8cc9-fcf85f287ef3" (UID: "058f82b5-15b1-4757-8cc9-fcf85f287ef3"). InnerVolumeSpecName "kube-api-access-l854b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:38:35.290477 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:35.290441    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/058f82b5-15b1-4757-8cc9-fcf85f287ef3-maas-api-tls" pod "058f82b5-15b1-4757-8cc9-fcf85f287ef3" (UID: "058f82b5-15b1-4757-8cc9-fcf85f287ef3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:38:35.381832 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:35.381799    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l854b\" (UniqueName: \"kubernetes.io/projected/058f82b5-15b1-4757-8cc9-fcf85f287ef3-kube-api-access-l854b\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:38:35.382031 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:35.382012    4431 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/058f82b5-15b1-4757-8cc9-fcf85f287ef3-metrics-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:38:35.382107 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:35.382035    4431 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/058f82b5-15b1-4757-8cc9-fcf85f287ef3-openshift-service-ca\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:38:35.382107 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:35.382052    4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/058f82b5-15b1-4757-8cc9-fcf85f287ef3-maas-api-tls\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:38:35.441818 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:35.441779    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:38:35.681239 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:35.680706    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:38:36.104003 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:36.103975    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8b1efeb6-98884dc8c-v2chc" event={"ID":"058f82b5-15b1-4757-8cc9-fcf85f287ef3","Type":"ContainerDied","Data":"3dcaa32a148277fe4054c96a9d1db9689bf091d8acbf4086f6b6adf17e73a44d"}
Nov 03 22:38:36.104139 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:36.104018    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8b1efeb6-98884dc8c-v2chc"
Nov 03 22:38:36.104139 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:36.104041    4431 scope.go:122] "RemoveContainer" containerID="cbf63fc7373cc42648112d74ccec6986216a33a37ce6053a512cd012205ee8bf"
Nov 03 22:38:36.162059 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:36.162023    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:38:36.162603 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:36.162567    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8b1efeb6-98884dc8c-v2chc"]
Nov 03 22:38:36.173476 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:36.173395    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8b1efeb6-98884dc8c-v2chc"]
Nov 03 22:38:37.549355 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:37.549318    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:38:38.089256 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:38.089206    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="058f82b5-15b1-4757-8cc9-fcf85f287ef3" path="/var/lib/kubelet/pods/058f82b5-15b1-4757-8cc9-fcf85f287ef3/volumes"
Nov 03 22:38:39.090904 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:39.090871    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:38:39.634508 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:39.634472    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:38:40.913162 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:40.913131    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:38:41.215897 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:41.215805    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:38:42.737130 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:42.737093    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:38:44.370565 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:44.370521    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:38:45.597314 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:45.597278    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-32fdf2-67d48ff746-m7rcv"]
Nov 03 22:38:45.597754 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:45.597556    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-32fdf2-67d48ff746-m7rcv" podUID="55062db9-46d7-4c21-8240-ee7853f114f8" containerName="maas-api" containerID="cri-o://4b18608700ea09bd0496204ab100d6d9091255b92df92b52747deaf01caf6b3f" gracePeriod=30
Nov 03 22:38:45.789166 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:45.788985    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:38:46.143142 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:46.143109    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-32fdf2-67d48ff746-m7rcv"
Nov 03 22:38:46.242687 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:46.242598    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55062db9-46d7-4c21-8240-ee7853f114f8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/55062db9-46d7-4c21-8240-ee7853f114f8-metrics-certs\") pod \"55062db9-46d7-4c21-8240-ee7853f114f8\" (UID: \"55062db9-46d7-4c21-8240-ee7853f114f8\") "
Nov 03 22:38:46.243073 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:46.243053    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55062db9-46d7-4c21-8240-ee7853f114f8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55062db9-46d7-4c21-8240-ee7853f114f8-maas-api-tls\") pod \"55062db9-46d7-4c21-8240-ee7853f114f8\" (UID: \"55062db9-46d7-4c21-8240-ee7853f114f8\") "
Nov 03 22:38:46.251284 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:46.243284    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55062db9-46d7-4c21-8240-ee7853f114f8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55062db9-46d7-4c21-8240-ee7853f114f8-openshift-service-ca\") pod \"55062db9-46d7-4c21-8240-ee7853f114f8\" (UID: \"55062db9-46d7-4c21-8240-ee7853f114f8\") "
Nov 03 22:38:46.252158 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:46.252135    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55062db9-46d7-4c21-8240-ee7853f114f8-kube-api-access-v9g8m\" (UniqueName: \"kubernetes.io/projected/55062db9-46d7-4c21-8240-ee7853f114f8-kube-api-access-v9g8m\") pod \"55062db9-46d7-4c21-8240-ee7853f114f8\" (UID: \"55062db9-46d7-4c21-8240-ee7853f114f8\") "
Nov 03 22:38:46.254635 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:46.251848    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55062db9-46d7-4c21-8240-ee7853f114f8-openshift-service-ca" pod "55062db9-46d7-4c21-8240-ee7853f114f8" (UID: "55062db9-46d7-4c21-8240-ee7853f114f8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:38:46.254791 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:46.254769    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55062db9-46d7-4c21-8240-ee7853f114f8-maas-api-tls" pod "55062db9-46d7-4c21-8240-ee7853f114f8" (UID: "55062db9-46d7-4c21-8240-ee7853f114f8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:38:46.256024 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:46.255707    4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55062db9-46d7-4c21-8240-ee7853f114f8-maas-api-tls\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:38:46.256024 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:46.255729    4431 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55062db9-46d7-4c21-8240-ee7853f114f8-openshift-service-ca\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:38:46.280598 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:46.276439    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55062db9-46d7-4c21-8240-ee7853f114f8-kube-api-access-v9g8m" pod "55062db9-46d7-4c21-8240-ee7853f114f8" (UID: "55062db9-46d7-4c21-8240-ee7853f114f8"). InnerVolumeSpecName "kube-api-access-v9g8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:38:46.283393 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:46.282711    4431 generic.go:356] "Generic (PLEG): container finished" podID="55062db9-46d7-4c21-8240-ee7853f114f8" containerID="4b18608700ea09bd0496204ab100d6d9091255b92df92b52747deaf01caf6b3f" exitCode=0
Nov 03 22:38:46.283393 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:46.282815    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-32fdf2-67d48ff746-m7rcv"
Nov 03 22:38:46.285792 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:46.285758    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-32fdf2-67d48ff746-m7rcv" event={"ID":"55062db9-46d7-4c21-8240-ee7853f114f8","Type":"ContainerDied","Data":"4b18608700ea09bd0496204ab100d6d9091255b92df92b52747deaf01caf6b3f"}
Nov 03 22:38:46.285935 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:46.285904    4431 scope.go:122] "RemoveContainer" containerID="4b18608700ea09bd0496204ab100d6d9091255b92df92b52747deaf01caf6b3f"
Nov 03 22:38:46.286016 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:46.285874    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-32fdf2-67d48ff746-m7rcv" event={"ID":"55062db9-46d7-4c21-8240-ee7853f114f8","Type":"ContainerDied","Data":"e16c627b1733e69b761ed2cc707c7a6bf46d9e19cc7fc4ffa9a5e90615306d67"}
Nov 03 22:38:46.349177 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:46.349146    4431 scope.go:122] "RemoveContainer" containerID="4b18608700ea09bd0496204ab100d6d9091255b92df92b52747deaf01caf6b3f"
Nov 03 22:38:46.349642 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:46.349618    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55062db9-46d7-4c21-8240-ee7853f114f8-metrics-certs" pod "55062db9-46d7-4c21-8240-ee7853f114f8" (UID: "55062db9-46d7-4c21-8240-ee7853f114f8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:38:46.351057 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:38:46.351030    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b18608700ea09bd0496204ab100d6d9091255b92df92b52747deaf01caf6b3f\": container with ID starting with 4b18608700ea09bd0496204ab100d6d9091255b92df92b52747deaf01caf6b3f not found: ID does not exist" containerID="4b18608700ea09bd0496204ab100d6d9091255b92df92b52747deaf01caf6b3f"
Nov 03 22:38:46.351403 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:46.351376    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b18608700ea09bd0496204ab100d6d9091255b92df92b52747deaf01caf6b3f"} err="failed to get container status \"4b18608700ea09bd0496204ab100d6d9091255b92df92b52747deaf01caf6b3f\": rpc error: code = NotFound desc = could not find container \"4b18608700ea09bd0496204ab100d6d9091255b92df92b52747deaf01caf6b3f\": container with ID starting with 4b18608700ea09bd0496204ab100d6d9091255b92df92b52747deaf01caf6b3f not found: ID does not exist"
Nov 03 22:38:46.352968 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:46.352947    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:38:46.382273 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:46.366203    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9g8m\" (UniqueName: \"kubernetes.io/projected/55062db9-46d7-4c21-8240-ee7853f114f8-kube-api-access-v9g8m\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:38:46.382273 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:46.380309    4431 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55062db9-46d7-4c21-8240-ee7853f114f8-metrics-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:38:46.528253 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:46.528144    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:38:46.629868 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:46.629838    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-32fdf2-67d48ff746-m7rcv"]
Nov 03 22:38:46.642399 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:46.642368    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-32fdf2-67d48ff746-m7rcv"]
Nov 03 22:38:46.954418 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:46.954383    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:38:47.178911 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:47.178873    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:38:48.089871 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:48.089839    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55062db9-46d7-4c21-8240-ee7853f114f8" path="/var/lib/kubelet/pods/55062db9-46d7-4c21-8240-ee7853f114f8/volumes"
Nov 03 22:38:48.188210 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:48.188173    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:38:49.862375 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:49.862342    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:38:51.234610 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:51.234564    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:38:51.379006 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:51.378977    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:38:51.518877 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:51.518794    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:38:52.216752 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:52.216711    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:38:53.027920 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:53.027873    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:38:53.751574 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:53.751546    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:38:53.902350 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:53.902309    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:38:54.387097 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:54.387062    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:38:55.095550 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:55.095515    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:38:56.875083 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:56.875007    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:38:57.098167 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:57.098113    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:38:57.418695 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:57.418666    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:38:57.623973 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:57.623921    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:38:57.809341 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:57.809256    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:38:58.181107 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:58.181068    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:38:59.690256 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:38:59.688509    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:39:01.260778 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:39:01.260744    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:39:01.434729 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:01.431559    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:39:03.005727 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:03.005686    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:39:03.724429 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:03.724393    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:39:04.645158 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:04.645123    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b1ec3a-6b45cb6cb5-d9lqz"]
Nov 03 22:39:04.646361 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:04.646333    4431 cpu_manager.go:414] "containerMap: removing container" podUID="22f9e7cf-d329-4178-af61-b9c27d485907" containerName="payload-processing"
Nov 03 22:39:04.646361 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:04.646359    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22f9e7cf-d329-4178-af61-b9c27d485907" containerName="payload-processing"
Nov 03 22:39:04.646572 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:04.646409    4431 cpu_manager.go:414] "containerMap: removing container" podUID="7e7cb728-8ff5-431d-80d8-87cffee7f726" containerName="payload-processing"
Nov 03 22:39:04.646572 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:04.646417    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e7cb728-8ff5-431d-80d8-87cffee7f726" containerName="payload-processing"
Nov 03 22:39:04.646572 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:04.646496    4431 cpu_manager.go:414] "containerMap: removing container" podUID="058f82b5-15b1-4757-8cc9-fcf85f287ef3" containerName="maas-api"
Nov 03 22:39:04.646572 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:04.646505    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="058f82b5-15b1-4757-8cc9-fcf85f287ef3" containerName="maas-api"
Nov 03 22:39:04.646572 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:04.646530    4431 cpu_manager.go:414] "containerMap: removing container" podUID="c907cfe8-b91c-4539-8503-20eb6561f3ad" containerName="payload-pre-processing"
Nov 03 22:39:04.646572 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:04.646538    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c907cfe8-b91c-4539-8503-20eb6561f3ad" containerName="payload-pre-processing"
Nov 03 22:39:04.646819 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:04.646579    4431 cpu_manager.go:414] "containerMap: removing container" podUID="55062db9-46d7-4c21-8240-ee7853f114f8" containerName="maas-api"
Nov 03 22:39:04.646819 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:04.646588    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55062db9-46d7-4c21-8240-ee7853f114f8" containerName="maas-api"
Nov 03 22:39:04.646819 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:04.646715    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="c907cfe8-b91c-4539-8503-20eb6561f3ad" containerName="payload-pre-processing"
Nov 03 22:39:04.646819 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:04.646733    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="55062db9-46d7-4c21-8240-ee7853f114f8" containerName="maas-api"
Nov 03 22:39:04.646819 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:04.646743    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="22f9e7cf-d329-4178-af61-b9c27d485907" containerName="payload-processing"
Nov 03 22:39:04.646819 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:04.646757    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e7cb728-8ff5-431d-80d8-87cffee7f726" containerName="payload-processing"
Nov 03 22:39:04.646819 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:04.646775    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="058f82b5-15b1-4757-8cc9-fcf85f287ef3" containerName="maas-api"
Nov 03 22:39:04.652494 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:04.652470    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b1ec3a-6b45cb6cb5-d9lqz"
Nov 03 22:39:04.684838 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:04.684800    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:39:04.685461 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:04.685434    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-b1ec3a-serving-cert\""
Nov 03 22:39:04.691012 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:04.690982    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b1ec3a-6b45cb6cb5-d9lqz"]
Nov 03 22:39:04.851028 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:04.839167    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a598ca79-668e-464c-ad39-5dd482218699-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b1ec3a-6b45cb6cb5-d9lqz\" (UID: \"a598ca79-668e-464c-ad39-5dd482218699\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b1ec3a-6b45cb6cb5-d9lqz"
Nov 03 22:39:04.851028 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:04.839403    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmcm7\" (UniqueName: \"kubernetes.io/projected/a598ca79-668e-464c-ad39-5dd482218699-kube-api-access-nmcm7\") pod \"maas-api-e2e-inf-a-b1ec3a-6b45cb6cb5-d9lqz\" (UID: \"a598ca79-668e-464c-ad39-5dd482218699\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b1ec3a-6b45cb6cb5-d9lqz"
Nov 03 22:39:04.851028 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:04.839566    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a598ca79-668e-464c-ad39-5dd482218699-metrics-certs\") pod \"maas-api-e2e-inf-a-b1ec3a-6b45cb6cb5-d9lqz\" (UID: \"a598ca79-668e-464c-ad39-5dd482218699\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b1ec3a-6b45cb6cb5-d9lqz"
Nov 03 22:39:04.851028 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:04.839637    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a598ca79-668e-464c-ad39-5dd482218699-maas-api-tls\") pod \"maas-api-e2e-inf-a-b1ec3a-6b45cb6cb5-d9lqz\" (UID: \"a598ca79-668e-464c-ad39-5dd482218699\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b1ec3a-6b45cb6cb5-d9lqz"
Nov 03 22:39:04.871593 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:04.871554    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:39:04.941540 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:04.941506    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a598ca79-668e-464c-ad39-5dd482218699-metrics-certs\") pod \"maas-api-e2e-inf-a-b1ec3a-6b45cb6cb5-d9lqz\" (UID: \"a598ca79-668e-464c-ad39-5dd482218699\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b1ec3a-6b45cb6cb5-d9lqz"
Nov 03 22:39:04.941748 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:04.941588    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a598ca79-668e-464c-ad39-5dd482218699-maas-api-tls\") pod \"maas-api-e2e-inf-a-b1ec3a-6b45cb6cb5-d9lqz\" (UID: \"a598ca79-668e-464c-ad39-5dd482218699\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b1ec3a-6b45cb6cb5-d9lqz"
Nov 03 22:39:04.941748 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:04.941689    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a598ca79-668e-464c-ad39-5dd482218699-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b1ec3a-6b45cb6cb5-d9lqz\" (UID: \"a598ca79-668e-464c-ad39-5dd482218699\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b1ec3a-6b45cb6cb5-d9lqz"
Nov 03 22:39:04.941903 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:04.941884    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmcm7\" (UniqueName: \"kubernetes.io/projected/a598ca79-668e-464c-ad39-5dd482218699-kube-api-access-nmcm7\") pod \"maas-api-e2e-inf-a-b1ec3a-6b45cb6cb5-d9lqz\" (UID: \"a598ca79-668e-464c-ad39-5dd482218699\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b1ec3a-6b45cb6cb5-d9lqz"
Nov 03 22:39:04.942763 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:04.942738    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a598ca79-668e-464c-ad39-5dd482218699-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b1ec3a-6b45cb6cb5-d9lqz\" (UID: \"a598ca79-668e-464c-ad39-5dd482218699\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b1ec3a-6b45cb6cb5-d9lqz"
Nov 03 22:39:04.945005 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:04.944980    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a598ca79-668e-464c-ad39-5dd482218699-maas-api-tls\") pod \"maas-api-e2e-inf-a-b1ec3a-6b45cb6cb5-d9lqz\" (UID: \"a598ca79-668e-464c-ad39-5dd482218699\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b1ec3a-6b45cb6cb5-d9lqz"
Nov 03 22:39:04.945695 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:04.945667    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a598ca79-668e-464c-ad39-5dd482218699-metrics-certs\") pod \"maas-api-e2e-inf-a-b1ec3a-6b45cb6cb5-d9lqz\" (UID: \"a598ca79-668e-464c-ad39-5dd482218699\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b1ec3a-6b45cb6cb5-d9lqz"
Nov 03 22:39:05.018685 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:05.018556    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmcm7\" (UniqueName: \"kubernetes.io/projected/a598ca79-668e-464c-ad39-5dd482218699-kube-api-access-nmcm7\") pod \"maas-api-e2e-inf-a-b1ec3a-6b45cb6cb5-d9lqz\" (UID: \"a598ca79-668e-464c-ad39-5dd482218699\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b1ec3a-6b45cb6cb5-d9lqz"
Nov 03 22:39:05.043788 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:05.043749    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:39:05.205443 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:05.205343    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:39:05.280237 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:05.276887    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b1ec3a-6b45cb6cb5-d9lqz"
Nov 03 22:39:05.539034 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:05.538768    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:39:05.665674 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:05.665651    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b1ec3a-6b45cb6cb5-d9lqz"]
Nov 03 22:39:05.667474 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:39:05.667440    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda598ca79_668e_464c_ad39_5dd482218699.slice/crio-bc2062aeb965932323f8ad35fb2ba79d118c390102475c2f7a9737331eae6d11 WatchSource:0}: Error finding container bc2062aeb965932323f8ad35fb2ba79d118c390102475c2f7a9737331eae6d11: Status 404 returned error can't find the container with id bc2062aeb965932323f8ad35fb2ba79d118c390102475c2f7a9737331eae6d11
Nov 03 22:39:05.681503 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:05.681451    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:39:05.887951 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:05.887868    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:39:06.500660 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:06.500619    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b1ec3a-6b45cb6cb5-d9lqz" event={"ID":"a598ca79-668e-464c-ad39-5dd482218699","Type":"ContainerStarted","Data":"13f61122fea2fac0aa1e42efdae1d183277370ee963abb86473f3cc61af8b292"}
Nov 03 22:39:06.500868 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:06.500685    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b1ec3a-6b45cb6cb5-d9lqz"
Nov 03 22:39:06.500868 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:06.500714    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b1ec3a-6b45cb6cb5-d9lqz" event={"ID":"a598ca79-668e-464c-ad39-5dd482218699","Type":"ContainerStarted","Data":"bc2062aeb965932323f8ad35fb2ba79d118c390102475c2f7a9737331eae6d11"}
Nov 03 22:39:06.544485 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:06.544430    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b1ec3a-6b45cb6cb5-d9lqz" podStartSLOduration=2.249127525 podStartE2EDuration="2.544415067s" totalImagesPullingTime="295.287542ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:39:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:39:06.543299873 +0000 UTC m=+1570.569155738" watchObservedRunningTime="2026-11-03 22:39:06.544415067 +0000 UTC m=+1570.570270930"
Nov 03 22:39:06.640766 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:06.640728    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:39:08.228799 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:08.228755    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:39:09.929535 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:09.929504    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:39:10.486795 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:10.486759    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:39:10.631041 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:10.630989    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:39:11.149820 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:11.149779    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:39:11.985825 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:11.985791    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq"]
Nov 03 22:39:11.986063 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:11.986034    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq" podUID="a49eb997-50b7-45cf-a401-19c1a2b7af77" containerName="istio-proxy" containerID="cri-o://fee49d0f08b5fdd10883faed8493c368ae1af1057309fe7a270c4a7d7a7f86cf" gracePeriod=30
Nov 03 22:39:12.517671 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:12.517629    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:39:12.523244 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:12.523185    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b1ec3a-6b45cb6cb5-d9lqz"
Nov 03 22:39:14.213971 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:14.213939    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:39:15.905533 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:15.905500    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:39:16.692177 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:16.692139    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:39:16.785488 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:16.785451    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm"]
Nov 03 22:39:16.785999 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:16.785733    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm" podUID="9b0aaae7-2a00-4487-a054-a359353bf4a9" containerName="istio-proxy" containerID="cri-o://1ba7b1aca193f6a8a973a82c01b6a19130b89e1f299d752d6e70676b6caf08a9" gracePeriod=30
Nov 03 22:39:16.852467 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:16.852431    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:39:17.010113 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:17.009739    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:39:17.336879 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:17.336790    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:39:17.438012 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:17.437981    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq"
Nov 03 22:39:17.487957 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:17.487554    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a49eb997-50b7-45cf-a401-19c1a2b7af77-istio-token\" (UniqueName: \"kubernetes.io/projected/a49eb997-50b7-45cf-a401-19c1a2b7af77-istio-token\") pod \"a49eb997-50b7-45cf-a401-19c1a2b7af77\" (UID: \"a49eb997-50b7-45cf-a401-19c1a2b7af77\") "
Nov 03 22:39:17.487957 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:17.487652    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a49eb997-50b7-45cf-a401-19c1a2b7af77-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a49eb997-50b7-45cf-a401-19c1a2b7af77-istiod-ca-cert\") pod \"a49eb997-50b7-45cf-a401-19c1a2b7af77\" (UID: \"a49eb997-50b7-45cf-a401-19c1a2b7af77\") "
Nov 03 22:39:17.487957 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:17.487691    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a49eb997-50b7-45cf-a401-19c1a2b7af77-kube-api-access-z2k8z\" (UniqueName: \"kubernetes.io/projected/a49eb997-50b7-45cf-a401-19c1a2b7af77-kube-api-access-z2k8z\") pod \"a49eb997-50b7-45cf-a401-19c1a2b7af77\" (UID: \"a49eb997-50b7-45cf-a401-19c1a2b7af77\") "
Nov 03 22:39:17.488334 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:17.488062    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a49eb997-50b7-45cf-a401-19c1a2b7af77-istiod-ca-cert" pod "a49eb997-50b7-45cf-a401-19c1a2b7af77" (UID: "a49eb997-50b7-45cf-a401-19c1a2b7af77"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:39:17.488396 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:17.488319    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a49eb997-50b7-45cf-a401-19c1a2b7af77-workload-certs" pod "a49eb997-50b7-45cf-a401-19c1a2b7af77" (UID: "a49eb997-50b7-45cf-a401-19c1a2b7af77"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:39:17.488396 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:17.487915    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a49eb997-50b7-45cf-a401-19c1a2b7af77-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a49eb997-50b7-45cf-a401-19c1a2b7af77-workload-certs\") pod \"a49eb997-50b7-45cf-a401-19c1a2b7af77\" (UID: \"a49eb997-50b7-45cf-a401-19c1a2b7af77\") "
Nov 03 22:39:17.488509 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:17.488456    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a49eb997-50b7-45cf-a401-19c1a2b7af77-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a49eb997-50b7-45cf-a401-19c1a2b7af77-credential-socket\") pod \"a49eb997-50b7-45cf-a401-19c1a2b7af77\" (UID: \"a49eb997-50b7-45cf-a401-19c1a2b7af77\") "
Nov 03 22:39:17.488784 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:17.488759    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a49eb997-50b7-45cf-a401-19c1a2b7af77-credential-socket" pod "a49eb997-50b7-45cf-a401-19c1a2b7af77" (UID: "a49eb997-50b7-45cf-a401-19c1a2b7af77"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:39:17.488898 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:17.488791    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a49eb997-50b7-45cf-a401-19c1a2b7af77-istio-data" pod "a49eb997-50b7-45cf-a401-19c1a2b7af77" (UID: "a49eb997-50b7-45cf-a401-19c1a2b7af77"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:39:17.488898 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:17.488490    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a49eb997-50b7-45cf-a401-19c1a2b7af77-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a49eb997-50b7-45cf-a401-19c1a2b7af77-istio-data\") pod \"a49eb997-50b7-45cf-a401-19c1a2b7af77\" (UID: \"a49eb997-50b7-45cf-a401-19c1a2b7af77\") "
Nov 03 22:39:17.489010 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:17.488934    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a49eb997-50b7-45cf-a401-19c1a2b7af77-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a49eb997-50b7-45cf-a401-19c1a2b7af77-workload-socket\") pod \"a49eb997-50b7-45cf-a401-19c1a2b7af77\" (UID: \"a49eb997-50b7-45cf-a401-19c1a2b7af77\") "
Nov 03 22:39:17.489010 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:17.488967    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a49eb997-50b7-45cf-a401-19c1a2b7af77-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a49eb997-50b7-45cf-a401-19c1a2b7af77-istio-podinfo\") pod \"a49eb997-50b7-45cf-a401-19c1a2b7af77\" (UID: \"a49eb997-50b7-45cf-a401-19c1a2b7af77\") "
Nov 03 22:39:17.489125 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:17.489040    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a49eb997-50b7-45cf-a401-19c1a2b7af77-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a49eb997-50b7-45cf-a401-19c1a2b7af77-istio-envoy\") pod \"a49eb997-50b7-45cf-a401-19c1a2b7af77\" (UID: \"a49eb997-50b7-45cf-a401-19c1a2b7af77\") "
Nov 03 22:39:17.489288 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:17.489266    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a49eb997-50b7-45cf-a401-19c1a2b7af77-workload-socket" pod "a49eb997-50b7-45cf-a401-19c1a2b7af77" (UID: "a49eb997-50b7-45cf-a401-19c1a2b7af77"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:39:17.489814 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:17.489789    4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a49eb997-50b7-45cf-a401-19c1a2b7af77-workload-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:39:17.489814 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:17.489812    4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a49eb997-50b7-45cf-a401-19c1a2b7af77-istiod-ca-cert\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:39:17.489814 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:17.489822    4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a49eb997-50b7-45cf-a401-19c1a2b7af77-workload-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:39:17.490054 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:17.489829    4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a49eb997-50b7-45cf-a401-19c1a2b7af77-credential-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:39:17.490054 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:17.489837    4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a49eb997-50b7-45cf-a401-19c1a2b7af77-istio-data\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:39:17.492486 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:17.492444    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a49eb997-50b7-45cf-a401-19c1a2b7af77-kube-api-access-z2k8z" pod "a49eb997-50b7-45cf-a401-19c1a2b7af77" (UID: "a49eb997-50b7-45cf-a401-19c1a2b7af77"). InnerVolumeSpecName "kube-api-access-z2k8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:39:17.492954 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:17.492930    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a49eb997-50b7-45cf-a401-19c1a2b7af77-istio-envoy" pod "a49eb997-50b7-45cf-a401-19c1a2b7af77" (UID: "a49eb997-50b7-45cf-a401-19c1a2b7af77"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:39:17.493667 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:17.493631    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a49eb997-50b7-45cf-a401-19c1a2b7af77-istio-token" pod "a49eb997-50b7-45cf-a401-19c1a2b7af77" (UID: "a49eb997-50b7-45cf-a401-19c1a2b7af77"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:39:17.494178 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:17.494160    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a49eb997-50b7-45cf-a401-19c1a2b7af77-istio-podinfo" pod "a49eb997-50b7-45cf-a401-19c1a2b7af77" (UID: "a49eb997-50b7-45cf-a401-19c1a2b7af77"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 22:39:17.583895 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:17.583799    4431 generic.go:356] "Generic (PLEG): container finished" podID="a49eb997-50b7-45cf-a401-19c1a2b7af77" containerID="fee49d0f08b5fdd10883faed8493c368ae1af1057309fe7a270c4a7d7a7f86cf" exitCode=0
Nov 03 22:39:17.583895 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:17.583855    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq"
Nov 03 22:39:17.583895 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:17.583886    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq" event={"ID":"a49eb997-50b7-45cf-a401-19c1a2b7af77","Type":"ContainerDied","Data":"fee49d0f08b5fdd10883faed8493c368ae1af1057309fe7a270c4a7d7a7f86cf"}
Nov 03 22:39:17.584195 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:17.583934    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq" event={"ID":"a49eb997-50b7-45cf-a401-19c1a2b7af77","Type":"ContainerDied","Data":"f3a43c7bdba36889a3f321e4422fcc1a44aa11192db249bc5bc8e1b4976cd271"}
Nov 03 22:39:17.584195 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:17.583961    4431 scope.go:122] "RemoveContainer" containerID="fee49d0f08b5fdd10883faed8493c368ae1af1057309fe7a270c4a7d7a7f86cf"
Nov 03 22:39:17.591310 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:17.591284    4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a49eb997-50b7-45cf-a401-19c1a2b7af77-istio-envoy\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:39:17.591310 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:17.591311    4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a49eb997-50b7-45cf-a401-19c1a2b7af77-istio-token\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:39:17.591502 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:17.591323    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2k8z\" (UniqueName: \"kubernetes.io/projected/a49eb997-50b7-45cf-a401-19c1a2b7af77-kube-api-access-z2k8z\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:39:17.591502 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:17.591336    4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a49eb997-50b7-45cf-a401-19c1a2b7af77-istio-podinfo\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:39:17.603201 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:17.603177    4431 scope.go:122] "RemoveContainer" containerID="fee49d0f08b5fdd10883faed8493c368ae1af1057309fe7a270c4a7d7a7f86cf"
Nov 03 22:39:17.603728 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:39:17.603692    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fee49d0f08b5fdd10883faed8493c368ae1af1057309fe7a270c4a7d7a7f86cf\": container with ID starting with fee49d0f08b5fdd10883faed8493c368ae1af1057309fe7a270c4a7d7a7f86cf not found: ID does not exist" containerID="fee49d0f08b5fdd10883faed8493c368ae1af1057309fe7a270c4a7d7a7f86cf"
Nov 03 22:39:17.603833 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:17.603741    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fee49d0f08b5fdd10883faed8493c368ae1af1057309fe7a270c4a7d7a7f86cf"} err="failed to get container status \"fee49d0f08b5fdd10883faed8493c368ae1af1057309fe7a270c4a7d7a7f86cf\": rpc error: code = NotFound desc = could not find container \"fee49d0f08b5fdd10883faed8493c368ae1af1057309fe7a270c4a7d7a7f86cf\": container with ID starting with fee49d0f08b5fdd10883faed8493c368ae1af1057309fe7a270c4a7d7a7f86cf not found: ID does not exist"
Nov 03 22:39:17.642752 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:17.642684    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq"]
Nov 03 22:39:17.646667 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:17.646636    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-v6bzq"]
Nov 03 22:39:17.843710 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:17.843625    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:39:18.018725 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:18.018684    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:39:18.092068 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:18.092039    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a49eb997-50b7-45cf-a401-19c1a2b7af77" path="/var/lib/kubelet/pods/a49eb997-50b7-45cf-a401-19c1a2b7af77/volumes"
Nov 03 22:39:19.328423 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:19.328390    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:39:19.552175 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:19.552134    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:39:19.742032 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:19.742001    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:39:20.388736 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:20.388706    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:39:21.249461 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:21.249431    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:39:21.488515 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:21.488481    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:39:21.688935 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:21.688894    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:39:22.250981 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:22.250953    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm"
Nov 03 22:39:22.351807 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:22.351721    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b0aaae7-2a00-4487-a054-a359353bf4a9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b0aaae7-2a00-4487-a054-a359353bf4a9-istiod-ca-cert\") pod \"9b0aaae7-2a00-4487-a054-a359353bf4a9\" (UID: \"9b0aaae7-2a00-4487-a054-a359353bf4a9\") "
Nov 03 22:39:22.351807 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:22.351804    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b0aaae7-2a00-4487-a054-a359353bf4a9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b0aaae7-2a00-4487-a054-a359353bf4a9-istio-data\") pod \"9b0aaae7-2a00-4487-a054-a359353bf4a9\" (UID: \"9b0aaae7-2a00-4487-a054-a359353bf4a9\") "
Nov 03 22:39:22.352064 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:22.351854    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b0aaae7-2a00-4487-a054-a359353bf4a9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b0aaae7-2a00-4487-a054-a359353bf4a9-workload-certs\") pod \"9b0aaae7-2a00-4487-a054-a359353bf4a9\" (UID: \"9b0aaae7-2a00-4487-a054-a359353bf4a9\") "
Nov 03 22:39:22.352064 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:22.351936    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b0aaae7-2a00-4487-a054-a359353bf4a9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b0aaae7-2a00-4487-a054-a359353bf4a9-credential-socket\") pod \"9b0aaae7-2a00-4487-a054-a359353bf4a9\" (UID: \"9b0aaae7-2a00-4487-a054-a359353bf4a9\") "
Nov 03 22:39:22.352064 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:22.351996    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b0aaae7-2a00-4487-a054-a359353bf4a9-istio-token\" (UniqueName: \"kubernetes.io/projected/9b0aaae7-2a00-4487-a054-a359353bf4a9-istio-token\") pod \"9b0aaae7-2a00-4487-a054-a359353bf4a9\" (UID: \"9b0aaae7-2a00-4487-a054-a359353bf4a9\") "
Nov 03 22:39:22.352247 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:22.352063    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9b0aaae7-2a00-4487-a054-a359353bf4a9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b0aaae7-2a00-4487-a054-a359353bf4a9-istio-podinfo\") pod \"9b0aaae7-2a00-4487-a054-a359353bf4a9\" (UID: \"9b0aaae7-2a00-4487-a054-a359353bf4a9\") "
Nov 03 22:39:22.352247 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:22.352090    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b0aaae7-2a00-4487-a054-a359353bf4a9-kube-api-access-545tj\" (UniqueName: \"kubernetes.io/projected/9b0aaae7-2a00-4487-a054-a359353bf4a9-kube-api-access-545tj\") pod \"9b0aaae7-2a00-4487-a054-a359353bf4a9\" (UID: \"9b0aaae7-2a00-4487-a054-a359353bf4a9\") "
Nov 03 22:39:22.352247 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:22.352124    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b0aaae7-2a00-4487-a054-a359353bf4a9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b0aaae7-2a00-4487-a054-a359353bf4a9-istio-envoy\") pod \"9b0aaae7-2a00-4487-a054-a359353bf4a9\" (UID: \"9b0aaae7-2a00-4487-a054-a359353bf4a9\") "
Nov 03 22:39:22.352247 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:22.352171    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b0aaae7-2a00-4487-a054-a359353bf4a9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b0aaae7-2a00-4487-a054-a359353bf4a9-workload-socket\") pod \"9b0aaae7-2a00-4487-a054-a359353bf4a9\" (UID: \"9b0aaae7-2a00-4487-a054-a359353bf4a9\") "
Nov 03 22:39:22.352473 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:22.352176    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b0aaae7-2a00-4487-a054-a359353bf4a9-istio-data" pod "9b0aaae7-2a00-4487-a054-a359353bf4a9" (UID: "9b0aaae7-2a00-4487-a054-a359353bf4a9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:39:22.352473 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:22.352279    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b0aaae7-2a00-4487-a054-a359353bf4a9-istiod-ca-cert" pod "9b0aaae7-2a00-4487-a054-a359353bf4a9" (UID: "9b0aaae7-2a00-4487-a054-a359353bf4a9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:39:22.352473 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:22.352285    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b0aaae7-2a00-4487-a054-a359353bf4a9-credential-socket" pod "9b0aaae7-2a00-4487-a054-a359353bf4a9" (UID: "9b0aaae7-2a00-4487-a054-a359353bf4a9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:39:22.352635 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:22.352504    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b0aaae7-2a00-4487-a054-a359353bf4a9-workload-socket" pod "9b0aaae7-2a00-4487-a054-a359353bf4a9" (UID: "9b0aaae7-2a00-4487-a054-a359353bf4a9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:39:22.353388 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:22.353119    4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b0aaae7-2a00-4487-a054-a359353bf4a9-istiod-ca-cert\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:39:22.353388 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:22.353146    4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b0aaae7-2a00-4487-a054-a359353bf4a9-istio-data\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:39:22.353388 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:22.353161    4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b0aaae7-2a00-4487-a054-a359353bf4a9-credential-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:39:22.353388 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:22.353176    4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b0aaae7-2a00-4487-a054-a359353bf4a9-workload-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:39:22.353673 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:22.353418    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b0aaae7-2a00-4487-a054-a359353bf4a9-workload-certs" pod "9b0aaae7-2a00-4487-a054-a359353bf4a9" (UID: "9b0aaae7-2a00-4487-a054-a359353bf4a9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:39:22.354833 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:22.354800    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b0aaae7-2a00-4487-a054-a359353bf4a9-kube-api-access-545tj" pod "9b0aaae7-2a00-4487-a054-a359353bf4a9" (UID: "9b0aaae7-2a00-4487-a054-a359353bf4a9"). InnerVolumeSpecName "kube-api-access-545tj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:39:22.354833 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:22.354823    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9b0aaae7-2a00-4487-a054-a359353bf4a9-istio-podinfo" pod "9b0aaae7-2a00-4487-a054-a359353bf4a9" (UID: "9b0aaae7-2a00-4487-a054-a359353bf4a9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 22:39:22.354991 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:22.354918    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b0aaae7-2a00-4487-a054-a359353bf4a9-istio-envoy" pod "9b0aaae7-2a00-4487-a054-a359353bf4a9" (UID: "9b0aaae7-2a00-4487-a054-a359353bf4a9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:39:22.355051 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:22.355038    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b0aaae7-2a00-4487-a054-a359353bf4a9-istio-token" pod "9b0aaae7-2a00-4487-a054-a359353bf4a9" (UID: "9b0aaae7-2a00-4487-a054-a359353bf4a9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:39:22.454888 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:22.454851    4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b0aaae7-2a00-4487-a054-a359353bf4a9-istio-token\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:39:22.454888 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:22.454886    4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b0aaae7-2a00-4487-a054-a359353bf4a9-istio-podinfo\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:39:22.454888 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:22.454897    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-545tj\" (UniqueName: \"kubernetes.io/projected/9b0aaae7-2a00-4487-a054-a359353bf4a9-kube-api-access-545tj\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:39:22.455129 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:22.454907    4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b0aaae7-2a00-4487-a054-a359353bf4a9-istio-envoy\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:39:22.455129 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:22.454916    4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b0aaae7-2a00-4487-a054-a359353bf4a9-workload-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:39:22.643326 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:22.643245    4431 generic.go:356] "Generic (PLEG): container finished" podID="9b0aaae7-2a00-4487-a054-a359353bf4a9" containerID="1ba7b1aca193f6a8a973a82c01b6a19130b89e1f299d752d6e70676b6caf08a9" exitCode=0
Nov 03 22:39:22.643326 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:22.643299    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm"
Nov 03 22:39:22.643326 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:22.643313    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm" event={"ID":"9b0aaae7-2a00-4487-a054-a359353bf4a9","Type":"ContainerDied","Data":"1ba7b1aca193f6a8a973a82c01b6a19130b89e1f299d752d6e70676b6caf08a9"}
Nov 03 22:39:22.643826 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:22.643356    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm" event={"ID":"9b0aaae7-2a00-4487-a054-a359353bf4a9","Type":"ContainerDied","Data":"70ee89ccd97634d8de551e74d8de8e2719fdbaf1773b3232f3f2ec5d82f67fa2"}
Nov 03 22:39:22.643826 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:22.643377    4431 scope.go:122] "RemoveContainer" containerID="1ba7b1aca193f6a8a973a82c01b6a19130b89e1f299d752d6e70676b6caf08a9"
Nov 03 22:39:22.654284 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:22.654266    4431 scope.go:122] "RemoveContainer" containerID="1ba7b1aca193f6a8a973a82c01b6a19130b89e1f299d752d6e70676b6caf08a9"
Nov 03 22:39:22.654581 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:39:22.654562    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ba7b1aca193f6a8a973a82c01b6a19130b89e1f299d752d6e70676b6caf08a9\": container with ID starting with 1ba7b1aca193f6a8a973a82c01b6a19130b89e1f299d752d6e70676b6caf08a9 not found: ID does not exist" containerID="1ba7b1aca193f6a8a973a82c01b6a19130b89e1f299d752d6e70676b6caf08a9"
Nov 03 22:39:22.654624 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:22.654590    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ba7b1aca193f6a8a973a82c01b6a19130b89e1f299d752d6e70676b6caf08a9"} err="failed to get container status \"1ba7b1aca193f6a8a973a82c01b6a19130b89e1f299d752d6e70676b6caf08a9\": rpc error: code = NotFound desc = could not find container \"1ba7b1aca193f6a8a973a82c01b6a19130b89e1f299d752d6e70676b6caf08a9\": container with ID starting with 1ba7b1aca193f6a8a973a82c01b6a19130b89e1f299d752d6e70676b6caf08a9 not found: ID does not exist"
Nov 03 22:39:22.725014 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:22.724924    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm"]
Nov 03 22:39:22.731826 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:22.731772    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-ztnxm"]
Nov 03 22:39:23.135032 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:23.134993    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:39:23.298958 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:23.298908    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:39:23.335809 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:23.335774    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9e540d-79df886dc8-q6kwt"]
Nov 03 22:39:23.336964 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:23.336936    4431 cpu_manager.go:414] "containerMap: removing container" podUID="9b0aaae7-2a00-4487-a054-a359353bf4a9" containerName="istio-proxy"
Nov 03 22:39:23.337172 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:23.337140    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b0aaae7-2a00-4487-a054-a359353bf4a9" containerName="istio-proxy"
Nov 03 22:39:23.337292 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:23.337235    4431 cpu_manager.go:414] "containerMap: removing container" podUID="a49eb997-50b7-45cf-a401-19c1a2b7af77" containerName="istio-proxy"
Nov 03 22:39:23.337292 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:23.337245    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a49eb997-50b7-45cf-a401-19c1a2b7af77" containerName="istio-proxy"
Nov 03 22:39:23.337454 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:23.337439    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b0aaae7-2a00-4487-a054-a359353bf4a9" containerName="istio-proxy"
Nov 03 22:39:23.337492 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:23.337467    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="a49eb997-50b7-45cf-a401-19c1a2b7af77" containerName="istio-proxy"
Nov 03 22:39:23.343499 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:23.343478    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9e540d-79df886dc8-q6kwt"
Nov 03 22:39:23.358124 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:23.358098    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9e540d-79df886dc8-q6kwt"]
Nov 03 22:39:23.422004 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:23.421978    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-9e540d\""
Nov 03 22:39:23.422151 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:23.422044    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-9e540d-dockercfg-dsvmc\""
Nov 03 22:39:23.461091 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:23.461055    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:39:23.490245 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:23.482411    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ec1402b-a0e6-4dc7-acf4-0644c9be7d81-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-9e540d-79df886dc8-q6kwt\" (UID: \"2ec1402b-a0e6-4dc7-acf4-0644c9be7d81\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9e540d-79df886dc8-q6kwt"
Nov 03 22:39:23.509635 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:23.499452    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7j57\" (UniqueName: \"kubernetes.io/projected/2ec1402b-a0e6-4dc7-acf4-0644c9be7d81-kube-api-access-b7j57\") pod \"payload-processing-e2e-inf-a-9e540d-79df886dc8-q6kwt\" (UID: \"2ec1402b-a0e6-4dc7-acf4-0644c9be7d81\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9e540d-79df886dc8-q6kwt"
Nov 03 22:39:23.615236 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:23.612634    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7j57\" (UniqueName: \"kubernetes.io/projected/2ec1402b-a0e6-4dc7-acf4-0644c9be7d81-kube-api-access-b7j57\") pod \"payload-processing-e2e-inf-a-9e540d-79df886dc8-q6kwt\" (UID: \"2ec1402b-a0e6-4dc7-acf4-0644c9be7d81\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9e540d-79df886dc8-q6kwt"
Nov 03 22:39:23.615236 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:23.612729    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ec1402b-a0e6-4dc7-acf4-0644c9be7d81-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-9e540d-79df886dc8-q6kwt\" (UID: \"2ec1402b-a0e6-4dc7-acf4-0644c9be7d81\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9e540d-79df886dc8-q6kwt"
Nov 03 22:39:23.615236 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:23.614048    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ec1402b-a0e6-4dc7-acf4-0644c9be7d81-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-9e540d-79df886dc8-q6kwt\" (UID: \"2ec1402b-a0e6-4dc7-acf4-0644c9be7d81\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9e540d-79df886dc8-q6kwt"
Nov 03 22:39:23.680134 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:23.680049    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7j57\" (UniqueName: \"kubernetes.io/projected/2ec1402b-a0e6-4dc7-acf4-0644c9be7d81-kube-api-access-b7j57\") pod \"payload-processing-e2e-inf-a-9e540d-79df886dc8-q6kwt\" (UID: \"2ec1402b-a0e6-4dc7-acf4-0644c9be7d81\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9e540d-79df886dc8-q6kwt"
Nov 03 22:39:23.961829 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:23.961706    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9e540d-79df886dc8-q6kwt"
Nov 03 22:39:24.093026 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:24.092994    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b0aaae7-2a00-4487-a054-a359353bf4a9" path="/var/lib/kubelet/pods/9b0aaae7-2a00-4487-a054-a359353bf4a9/volumes"
Nov 03 22:39:24.294983 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:24.294954    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9e540d-79df886dc8-q6kwt"]
Nov 03 22:39:24.295601 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:39:24.295576    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ec1402b_a0e6_4dc7_acf4_0644c9be7d81.slice/crio-0cd3543ef7a9bf91983426ac42ada9b9fdfd9df6e4ce43d95ed770e73a997a79 WatchSource:0}: Error finding container 0cd3543ef7a9bf91983426ac42ada9b9fdfd9df6e4ce43d95ed770e73a997a79: Status 404 returned error can't find the container with id 0cd3543ef7a9bf91983426ac42ada9b9fdfd9df6e4ce43d95ed770e73a997a79
Nov 03 22:39:24.684068 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:24.684028    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9e540d-79df886dc8-q6kwt" event={"ID":"2ec1402b-a0e6-4dc7-acf4-0644c9be7d81","Type":"ContainerStarted","Data":"1d19a42624a4d40b2d553b49b767267fdcc9ee4ed3ae7b861fc8786670bbeb81"}
Nov 03 22:39:24.684550 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:24.684090    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9e540d-79df886dc8-q6kwt" event={"ID":"2ec1402b-a0e6-4dc7-acf4-0644c9be7d81","Type":"ContainerStarted","Data":"0cd3543ef7a9bf91983426ac42ada9b9fdfd9df6e4ce43d95ed770e73a997a79"}
Nov 03 22:39:24.684550 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:24.684120    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-9e540d-79df886dc8-q6kwt"
Nov 03 22:39:24.739807 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:24.739751    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-9e540d-79df886dc8-q6kwt" podStartSLOduration=1.7397372789999999 podStartE2EDuration="1.739737279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:39:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:39:24.736608455 +0000 UTC m=+1588.762464316" watchObservedRunningTime="2026-11-03 22:39:24.739737279 +0000 UTC m=+1588.765593180"
Nov 03 22:39:25.334974 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:25.334942    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:39:26.679318 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:26.679279    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:39:26.850615 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:26.850580    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:39:27.018310 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:27.018202    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:39:27.190236 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:27.189849    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:39:28.625660 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:28.625626    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:39:30.246560 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:30.246515    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:39:30.893986 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:30.893946    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:39:31.051475 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:31.051439    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:39:31.260472 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:39:31.260437    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:39:31.994518 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:31.994485    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:39:33.654281 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:33.654244    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:39:34.279783 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:34.279752    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:39:34.423135 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:34.423102    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:39:35.446307 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:35.446270    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:39:35.694321 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:35.694290    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-9e540d-79df886dc8-q6kwt"
Nov 03 22:39:37.124020 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:37.123983    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:39:38.721415 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:38.721379    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:39:40.252920 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:40.252755    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:39:41.928639 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:41.928600    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:39:43.792366 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:43.792325    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:39:45.733454 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:45.733412    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:39:46.356801 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:46.356765    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:39:46.868611 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:46.868574    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:39:47.371471 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:47.371436    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:39:47.625914 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:47.625775    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:39:47.791776 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:47.791741    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:39:49.008896 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:49.008837    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:39:49.549521 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:49.549485    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:39:49.693432 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:49.693401    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:39:49.847004 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:49.846922    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:39:50.204864 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:50.204831    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:39:51.284663 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:51.284628    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:39:51.477546 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:51.477513    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:39:51.626336 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:51.626254    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:39:52.340847 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:52.340811    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:39:52.913699 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:52.913671    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:39:53.065002 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:53.064961    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:39:53.260053 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:53.259899    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:39:54.859007 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:54.858889    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:39:56.645487 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:56.645438    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:39:56.791497 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:56.791463    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:39:57.775002 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:57.774963    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:39:57.940513 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:57.940473    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:39:58.158449 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:58.158367    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:39:58.455997 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:39:58.455960    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:40:00.307453 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:00.307208    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:40:01.264267 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:40:01.264207    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:40:01.925776 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:01.921927    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:40:03.613988 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:03.613953    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:40:05.225849 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:05.225802    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:40:06.900510 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:06.900470    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:40:08.723245 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:08.723188    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:40:09.617232 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:09.617191    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:40:09.771103 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:09.771070    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:40:10.839963 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:10.839928    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:40:13.152073 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:13.152022    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:40:15.096859 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:15.096814    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:40:16.910980 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:16.910918    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:40:16.950028 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:16.949984    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv"]
Nov 03 22:40:16.950395 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:16.950356    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv" podUID="27596c43-8204-4fd0-9ab0-612997367c40" containerName="istio-proxy" containerID="cri-o://7724ccc321cb6f5134ba8959602ea50740164befaddf398b8ead2a4ae94de574" gracePeriod=30
Nov 03 22:40:18.220345 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:18.220302    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:40:18.761415 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:18.761370    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:40:18.998337 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:18.998302    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:40:19.143520 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:19.143430    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:40:19.429943 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:19.429904    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:40:20.678187 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:20.678153    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:40:22.197770 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:22.197734    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:40:22.221028 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:22.220987    4431 generic.go:356] "Generic (PLEG): container finished" podID="27596c43-8204-4fd0-9ab0-612997367c40" containerID="7724ccc321cb6f5134ba8959602ea50740164befaddf398b8ead2a4ae94de574" exitCode=0
Nov 03 22:40:22.221235 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:22.221119    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv" event={"ID":"27596c43-8204-4fd0-9ab0-612997367c40","Type":"ContainerDied","Data":"7724ccc321cb6f5134ba8959602ea50740164befaddf398b8ead2a4ae94de574"}
Nov 03 22:40:22.264031 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:22.263868    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv"
Nov 03 22:40:22.270992 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:22.270941    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27596c43-8204-4fd0-9ab0-612997367c40-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27596c43-8204-4fd0-9ab0-612997367c40-istiod-ca-cert\") pod \"27596c43-8204-4fd0-9ab0-612997367c40\" (UID: \"27596c43-8204-4fd0-9ab0-612997367c40\") "
Nov 03 22:40:22.272476 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:22.272421    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27596c43-8204-4fd0-9ab0-612997367c40-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27596c43-8204-4fd0-9ab0-612997367c40-workload-socket\") pod \"27596c43-8204-4fd0-9ab0-612997367c40\" (UID: \"27596c43-8204-4fd0-9ab0-612997367c40\") "
Nov 03 22:40:22.272739 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:22.271781    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27596c43-8204-4fd0-9ab0-612997367c40-istiod-ca-cert" pod "27596c43-8204-4fd0-9ab0-612997367c40" (UID: "27596c43-8204-4fd0-9ab0-612997367c40"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:40:22.273738 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:22.273584    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27596c43-8204-4fd0-9ab0-612997367c40-workload-socket" pod "27596c43-8204-4fd0-9ab0-612997367c40" (UID: "27596c43-8204-4fd0-9ab0-612997367c40"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:40:22.274060 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:22.272909    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27596c43-8204-4fd0-9ab0-612997367c40-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27596c43-8204-4fd0-9ab0-612997367c40-credential-socket\") pod \"27596c43-8204-4fd0-9ab0-612997367c40\" (UID: \"27596c43-8204-4fd0-9ab0-612997367c40\") "
Nov 03 22:40:22.274343 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:22.274324    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27596c43-8204-4fd0-9ab0-612997367c40-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27596c43-8204-4fd0-9ab0-612997367c40-workload-certs\") pod \"27596c43-8204-4fd0-9ab0-612997367c40\" (UID: \"27596c43-8204-4fd0-9ab0-612997367c40\") "
Nov 03 22:40:22.277396 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:22.277124    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27596c43-8204-4fd0-9ab0-612997367c40-workload-certs" pod "27596c43-8204-4fd0-9ab0-612997367c40" (UID: "27596c43-8204-4fd0-9ab0-612997367c40"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:40:22.278044 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:22.277992    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/27596c43-8204-4fd0-9ab0-612997367c40-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27596c43-8204-4fd0-9ab0-612997367c40-istio-podinfo\") pod \"27596c43-8204-4fd0-9ab0-612997367c40\" (UID: \"27596c43-8204-4fd0-9ab0-612997367c40\") "
Nov 03 22:40:22.278477 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:22.278457    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27596c43-8204-4fd0-9ab0-612997367c40-istio-token\" (UniqueName: \"kubernetes.io/projected/27596c43-8204-4fd0-9ab0-612997367c40-istio-token\") pod \"27596c43-8204-4fd0-9ab0-612997367c40\" (UID: \"27596c43-8204-4fd0-9ab0-612997367c40\") "
Nov 03 22:40:22.281422 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:22.281388    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27596c43-8204-4fd0-9ab0-612997367c40-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27596c43-8204-4fd0-9ab0-612997367c40-istio-envoy\") pod \"27596c43-8204-4fd0-9ab0-612997367c40\" (UID: \"27596c43-8204-4fd0-9ab0-612997367c40\") "
Nov 03 22:40:22.284362 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:22.284341    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27596c43-8204-4fd0-9ab0-612997367c40-kube-api-access-69gkv\" (UniqueName: \"kubernetes.io/projected/27596c43-8204-4fd0-9ab0-612997367c40-kube-api-access-69gkv\") pod \"27596c43-8204-4fd0-9ab0-612997367c40\" (UID: \"27596c43-8204-4fd0-9ab0-612997367c40\") "
Nov 03 22:40:22.285075 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:22.285039    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27596c43-8204-4fd0-9ab0-612997367c40-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27596c43-8204-4fd0-9ab0-612997367c40-istio-data\") pod \"27596c43-8204-4fd0-9ab0-612997367c40\" (UID: \"27596c43-8204-4fd0-9ab0-612997367c40\") "
Nov 03 22:40:22.286443 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:22.286329    4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27596c43-8204-4fd0-9ab0-612997367c40-istiod-ca-cert\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:40:22.286600 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:22.286585    4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27596c43-8204-4fd0-9ab0-612997367c40-workload-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:40:22.286730 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:22.286716    4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27596c43-8204-4fd0-9ab0-612997367c40-workload-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:40:22.286858 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:22.281199    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27596c43-8204-4fd0-9ab0-612997367c40-credential-socket" pod "27596c43-8204-4fd0-9ab0-612997367c40" (UID: "27596c43-8204-4fd0-9ab0-612997367c40"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:40:22.288860 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:22.285433    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27596c43-8204-4fd0-9ab0-612997367c40-istio-data" pod "27596c43-8204-4fd0-9ab0-612997367c40" (UID: "27596c43-8204-4fd0-9ab0-612997367c40"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:40:22.288996 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:22.288788    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27596c43-8204-4fd0-9ab0-612997367c40-kube-api-access-69gkv" pod "27596c43-8204-4fd0-9ab0-612997367c40" (UID: "27596c43-8204-4fd0-9ab0-612997367c40"). InnerVolumeSpecName "kube-api-access-69gkv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:40:22.289431 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:22.289395    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/27596c43-8204-4fd0-9ab0-612997367c40-istio-podinfo" pod "27596c43-8204-4fd0-9ab0-612997367c40" (UID: "27596c43-8204-4fd0-9ab0-612997367c40"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 22:40:22.289520 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:22.289471    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27596c43-8204-4fd0-9ab0-612997367c40-istio-envoy" pod "27596c43-8204-4fd0-9ab0-612997367c40" (UID: "27596c43-8204-4fd0-9ab0-612997367c40"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:40:22.290763 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:22.290742    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27596c43-8204-4fd0-9ab0-612997367c40-istio-token" pod "27596c43-8204-4fd0-9ab0-612997367c40" (UID: "27596c43-8204-4fd0-9ab0-612997367c40"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:40:22.388772 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:22.388736    4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27596c43-8204-4fd0-9ab0-612997367c40-istio-data\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:40:22.388772 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:22.388768    4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27596c43-8204-4fd0-9ab0-612997367c40-credential-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:40:22.388772 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:22.388778    4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27596c43-8204-4fd0-9ab0-612997367c40-istio-podinfo\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:40:22.389028 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:22.388786    4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27596c43-8204-4fd0-9ab0-612997367c40-istio-token\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:40:22.389028 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:22.388794    4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27596c43-8204-4fd0-9ab0-612997367c40-istio-envoy\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:40:22.389028 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:22.388802    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69gkv\" (UniqueName: \"kubernetes.io/projected/27596c43-8204-4fd0-9ab0-612997367c40-kube-api-access-69gkv\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:40:23.190154 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:23.190121    4431 patch_prober.go:29] interesting pod/e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 22:40:23.190384 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:23.190184    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv" podUID="27596c43-8204-4fd0-9ab0-612997367c40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:40:23.229065 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:23.229035    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv"
Nov 03 22:40:23.229541 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:23.229058    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv" event={"ID":"27596c43-8204-4fd0-9ab0-612997367c40","Type":"ContainerDied","Data":"108d8c05dab33f5bfc5d0b0539fca223bdc6a007194c16baa01e1c4f12c593eb"}
Nov 03 22:40:23.229541 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:23.229121    4431 scope.go:122] "RemoveContainer" containerID="7724ccc321cb6f5134ba8959602ea50740164befaddf398b8ead2a4ae94de574"
Nov 03 22:40:23.295580 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:23.295545    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv"]
Nov 03 22:40:23.309650 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:23.309612    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-7pkkv"]
Nov 03 22:40:23.871770 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:23.871712    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:40:24.051787 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:24.051748    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:40:24.088988 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:24.088957    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27596c43-8204-4fd0-9ab0-612997367c40" path="/var/lib/kubelet/pods/27596c43-8204-4fd0-9ab0-612997367c40/volumes"
Nov 03 22:40:25.683770 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:25.683729    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:40:27.024864 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:27.024820    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:40:27.200208 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:27.200172    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:40:27.373842 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:27.373751    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:40:28.142827 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:28.142792    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:40:28.711241 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:28.700617    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:40:29.077440 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:29.077357    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:40:29.231035 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:29.230991    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:40:29.386288 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:29.386137    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:40:31.014180 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:31.014149    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:40:31.261353 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:40:31.261315    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:40:31.411010 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:31.410912    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9fdlh"]
Nov 03 22:40:31.415243 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:31.411457    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9fdlh" podUID="9a625b54-3dd0-4a42-9583-fe46b592c1a6" containerName="istio-proxy" containerID="cri-o://fbcd896eba36a903dbcf4a1e2284b12c4a6807dd92b7b707f876eadafa858038" gracePeriod=30
Nov 03 22:40:31.467381 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:31.467293    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-85x5q"]
Nov 03 22:40:31.468096 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:31.468010    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-85x5q" podUID="6dffba47-88ed-4357-862e-d5c6e1d34060" containerName="istio-proxy" containerID="cri-o://68a696bb0f8bec8e92449bc8c6ac53a1e84dcbd6fb8657cf4009a1df8f49428b" gracePeriod=30
Nov 03 22:40:32.003405 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:32.003373    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf"]
Nov 03 22:40:32.003610 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:32.003585    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf" podUID="4db918b5-a69e-45f7-996a-2658132cbffc" containerName="istio-proxy" containerID="cri-o://3e2220b8c700b90cf1c7eff5571b93b90a3c2f6031f4c061ca140100387d0e73" gracePeriod=30
Nov 03 22:40:32.896608 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:32.896565    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:40:33.214240 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:33.214193    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:40:33.390010 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:33.389967    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:40:34.710462 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:34.710416    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:40:34.887079 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:34.887051    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:40:36.049051 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:36.049017    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:40:36.676274 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:36.675854    4431 patch_prober.go:29] interesting pod/e2e-worker-w2-553753-openshift-default-558949869c-85x5q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 22:40:36.676274 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:36.675927    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-85x5q" podUID="6dffba47-88ed-4357-862e-d5c6e1d34060" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 22:40:36.760342 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:36.759921    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:40:36.879206 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:36.878502    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9fdlh"
Nov 03 22:40:36.903082 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:36.903049    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a625b54-3dd0-4a42-9583-fe46b592c1a6-istio-token\" (UniqueName: \"kubernetes.io/projected/9a625b54-3dd0-4a42-9583-fe46b592c1a6-istio-token\") pod \"9a625b54-3dd0-4a42-9583-fe46b592c1a6\" (UID: \"9a625b54-3dd0-4a42-9583-fe46b592c1a6\") "
Nov 03 22:40:36.903263 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:36.903183    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a625b54-3dd0-4a42-9583-fe46b592c1a6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a625b54-3dd0-4a42-9583-fe46b592c1a6-workload-socket\") pod \"9a625b54-3dd0-4a42-9583-fe46b592c1a6\" (UID: \"9a625b54-3dd0-4a42-9583-fe46b592c1a6\") "
Nov 03 22:40:36.903326 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:36.903268    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a625b54-3dd0-4a42-9583-fe46b592c1a6-kube-api-access-ngwjt\" (UniqueName: \"kubernetes.io/projected/9a625b54-3dd0-4a42-9583-fe46b592c1a6-kube-api-access-ngwjt\") pod \"9a625b54-3dd0-4a42-9583-fe46b592c1a6\" (UID: \"9a625b54-3dd0-4a42-9583-fe46b592c1a6\") "
Nov 03 22:40:36.903372 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:36.903322    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a625b54-3dd0-4a42-9583-fe46b592c1a6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a625b54-3dd0-4a42-9583-fe46b592c1a6-istio-envoy\") pod \"9a625b54-3dd0-4a42-9583-fe46b592c1a6\" (UID: \"9a625b54-3dd0-4a42-9583-fe46b592c1a6\") "
Nov 03 22:40:36.903372 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:36.903357    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a625b54-3dd0-4a42-9583-fe46b592c1a6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a625b54-3dd0-4a42-9583-fe46b592c1a6-credential-socket\") pod \"9a625b54-3dd0-4a42-9583-fe46b592c1a6\" (UID: \"9a625b54-3dd0-4a42-9583-fe46b592c1a6\") "
Nov 03 22:40:36.903469 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:36.903446    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a625b54-3dd0-4a42-9583-fe46b592c1a6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a625b54-3dd0-4a42-9583-fe46b592c1a6-istio-data\") pod \"9a625b54-3dd0-4a42-9583-fe46b592c1a6\" (UID: \"9a625b54-3dd0-4a42-9583-fe46b592c1a6\") "
Nov 03 22:40:36.903792 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:36.903502    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a625b54-3dd0-4a42-9583-fe46b592c1a6-workload-socket" pod "9a625b54-3dd0-4a42-9583-fe46b592c1a6" (UID: "9a625b54-3dd0-4a42-9583-fe46b592c1a6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:40:36.903792 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:36.903513    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a625b54-3dd0-4a42-9583-fe46b592c1a6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a625b54-3dd0-4a42-9583-fe46b592c1a6-istio-podinfo\") pod \"9a625b54-3dd0-4a42-9583-fe46b592c1a6\" (UID: \"9a625b54-3dd0-4a42-9583-fe46b592c1a6\") "
Nov 03 22:40:36.903792 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:36.903561    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a625b54-3dd0-4a42-9583-fe46b592c1a6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a625b54-3dd0-4a42-9583-fe46b592c1a6-istiod-ca-cert\") pod \"9a625b54-3dd0-4a42-9583-fe46b592c1a6\" (UID: \"9a625b54-3dd0-4a42-9583-fe46b592c1a6\") "
Nov 03 22:40:36.903792 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:36.903628    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a625b54-3dd0-4a42-9583-fe46b592c1a6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a625b54-3dd0-4a42-9583-fe46b592c1a6-workload-certs\") pod \"9a625b54-3dd0-4a42-9583-fe46b592c1a6\" (UID: \"9a625b54-3dd0-4a42-9583-fe46b592c1a6\") "
Nov 03 22:40:36.903792 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:36.903724    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a625b54-3dd0-4a42-9583-fe46b592c1a6-credential-socket" pod "9a625b54-3dd0-4a42-9583-fe46b592c1a6" (UID: "9a625b54-3dd0-4a42-9583-fe46b592c1a6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:40:36.904099 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:36.903809    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a625b54-3dd0-4a42-9583-fe46b592c1a6-istio-data" pod "9a625b54-3dd0-4a42-9583-fe46b592c1a6" (UID: "9a625b54-3dd0-4a42-9583-fe46b592c1a6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:40:36.904099 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:36.903998    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a625b54-3dd0-4a42-9583-fe46b592c1a6-istiod-ca-cert" pod "9a625b54-3dd0-4a42-9583-fe46b592c1a6" (UID: "9a625b54-3dd0-4a42-9583-fe46b592c1a6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:40:36.904099 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:36.904077    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a625b54-3dd0-4a42-9583-fe46b592c1a6-workload-certs" pod "9a625b54-3dd0-4a42-9583-fe46b592c1a6" (UID: "9a625b54-3dd0-4a42-9583-fe46b592c1a6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:40:36.904668 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:36.904574    4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a625b54-3dd0-4a42-9583-fe46b592c1a6-workload-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:40:36.904668 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:36.904599    4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a625b54-3dd0-4a42-9583-fe46b592c1a6-credential-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:40:36.904668 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:36.904614    4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a625b54-3dd0-4a42-9583-fe46b592c1a6-istio-data\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:40:36.904668 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:36.904627    4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a625b54-3dd0-4a42-9583-fe46b592c1a6-istiod-ca-cert\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:40:36.904668 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:36.904638    4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a625b54-3dd0-4a42-9583-fe46b592c1a6-workload-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:40:36.905596 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:36.905575    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a625b54-3dd0-4a42-9583-fe46b592c1a6-istio-podinfo" pod "9a625b54-3dd0-4a42-9583-fe46b592c1a6" (UID: "9a625b54-3dd0-4a42-9583-fe46b592c1a6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 22:40:36.905824 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:36.905808    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a625b54-3dd0-4a42-9583-fe46b592c1a6-istio-token" pod "9a625b54-3dd0-4a42-9583-fe46b592c1a6" (UID: "9a625b54-3dd0-4a42-9583-fe46b592c1a6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:40:36.906357 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:36.906334    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a625b54-3dd0-4a42-9583-fe46b592c1a6-kube-api-access-ngwjt" pod "9a625b54-3dd0-4a42-9583-fe46b592c1a6" (UID: "9a625b54-3dd0-4a42-9583-fe46b592c1a6"). InnerVolumeSpecName "kube-api-access-ngwjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:40:36.906735 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:36.906718    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a625b54-3dd0-4a42-9583-fe46b592c1a6-istio-envoy" pod "9a625b54-3dd0-4a42-9583-fe46b592c1a6" (UID: "9a625b54-3dd0-4a42-9583-fe46b592c1a6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:40:36.917286 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:36.917230    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:40:36.929730 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:36.929709    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-85x5q"
Nov 03 22:40:37.005456 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.005422    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6dffba47-88ed-4357-862e-d5c6e1d34060-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dffba47-88ed-4357-862e-d5c6e1d34060-istio-podinfo\") pod \"6dffba47-88ed-4357-862e-d5c6e1d34060\" (UID: \"6dffba47-88ed-4357-862e-d5c6e1d34060\") "
Nov 03 22:40:37.005628 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.005580    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dffba47-88ed-4357-862e-d5c6e1d34060-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dffba47-88ed-4357-862e-d5c6e1d34060-credential-socket\") pod \"6dffba47-88ed-4357-862e-d5c6e1d34060\" (UID: \"6dffba47-88ed-4357-862e-d5c6e1d34060\") "
Nov 03 22:40:37.005696 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.005681    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dffba47-88ed-4357-862e-d5c6e1d34060-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dffba47-88ed-4357-862e-d5c6e1d34060-istio-envoy\") pod \"6dffba47-88ed-4357-862e-d5c6e1d34060\" (UID: \"6dffba47-88ed-4357-862e-d5c6e1d34060\") "
Nov 03 22:40:37.005748 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.005722    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dffba47-88ed-4357-862e-d5c6e1d34060-kube-api-access-qm4cj\" (UniqueName: \"kubernetes.io/projected/6dffba47-88ed-4357-862e-d5c6e1d34060-kube-api-access-qm4cj\") pod \"6dffba47-88ed-4357-862e-d5c6e1d34060\" (UID: \"6dffba47-88ed-4357-862e-d5c6e1d34060\") "
Nov 03 22:40:37.005832 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.005817    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dffba47-88ed-4357-862e-d5c6e1d34060-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dffba47-88ed-4357-862e-d5c6e1d34060-workload-certs\") pod \"6dffba47-88ed-4357-862e-d5c6e1d34060\" (UID: \"6dffba47-88ed-4357-862e-d5c6e1d34060\") "
Nov 03 22:40:37.005920 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.005896    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dffba47-88ed-4357-862e-d5c6e1d34060-credential-socket" pod "6dffba47-88ed-4357-862e-d5c6e1d34060" (UID: "6dffba47-88ed-4357-862e-d5c6e1d34060"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:40:37.005980 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.005924    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6dffba47-88ed-4357-862e-d5c6e1d34060-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dffba47-88ed-4357-862e-d5c6e1d34060-istiod-ca-cert\") pod \"6dffba47-88ed-4357-862e-d5c6e1d34060\" (UID: \"6dffba47-88ed-4357-862e-d5c6e1d34060\") "
Nov 03 22:40:37.006035 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.005978    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dffba47-88ed-4357-862e-d5c6e1d34060-istio-token\" (UniqueName: \"kubernetes.io/projected/6dffba47-88ed-4357-862e-d5c6e1d34060-istio-token\") pod \"6dffba47-88ed-4357-862e-d5c6e1d34060\" (UID: \"6dffba47-88ed-4357-862e-d5c6e1d34060\") "
Nov 03 22:40:37.006035 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.006020    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dffba47-88ed-4357-862e-d5c6e1d34060-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dffba47-88ed-4357-862e-d5c6e1d34060-workload-socket\") pod \"6dffba47-88ed-4357-862e-d5c6e1d34060\" (UID: \"6dffba47-88ed-4357-862e-d5c6e1d34060\") "
Nov 03 22:40:37.006157 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.006056    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dffba47-88ed-4357-862e-d5c6e1d34060-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dffba47-88ed-4357-862e-d5c6e1d34060-istio-data\") pod \"6dffba47-88ed-4357-862e-d5c6e1d34060\" (UID: \"6dffba47-88ed-4357-862e-d5c6e1d34060\") "
Nov 03 22:40:37.006157 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.006116    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dffba47-88ed-4357-862e-d5c6e1d34060-workload-certs" pod "6dffba47-88ed-4357-862e-d5c6e1d34060" (UID: "6dffba47-88ed-4357-862e-d5c6e1d34060"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:40:37.006385 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.006360    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dffba47-88ed-4357-862e-d5c6e1d34060-workload-socket" pod "6dffba47-88ed-4357-862e-d5c6e1d34060" (UID: "6dffba47-88ed-4357-862e-d5c6e1d34060"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:40:37.006385 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.006370    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dffba47-88ed-4357-862e-d5c6e1d34060-istiod-ca-cert" pod "6dffba47-88ed-4357-862e-d5c6e1d34060" (UID: "6dffba47-88ed-4357-862e-d5c6e1d34060"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:40:37.006604 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.006575    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dffba47-88ed-4357-862e-d5c6e1d34060-istio-data" pod "6dffba47-88ed-4357-862e-d5c6e1d34060" (UID: "6dffba47-88ed-4357-862e-d5c6e1d34060"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:40:37.007006 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.006986    4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a625b54-3dd0-4a42-9583-fe46b592c1a6-istio-podinfo\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:40:37.007104 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.007012    4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dffba47-88ed-4357-862e-d5c6e1d34060-workload-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:40:37.007104 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.007027    4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a625b54-3dd0-4a42-9583-fe46b592c1a6-istio-token\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:40:37.007104 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.007040    4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dffba47-88ed-4357-862e-d5c6e1d34060-istiod-ca-cert\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:40:37.007104 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.007053    4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dffba47-88ed-4357-862e-d5c6e1d34060-workload-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:40:37.007104 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.007066    4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dffba47-88ed-4357-862e-d5c6e1d34060-istio-data\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:40:37.007104 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.007082    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngwjt\" (UniqueName: \"kubernetes.io/projected/9a625b54-3dd0-4a42-9583-fe46b592c1a6-kube-api-access-ngwjt\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:40:37.007104 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.007097    4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a625b54-3dd0-4a42-9583-fe46b592c1a6-istio-envoy\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:40:37.007469 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.007110    4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dffba47-88ed-4357-862e-d5c6e1d34060-credential-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:40:37.008005 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.007981    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dffba47-88ed-4357-862e-d5c6e1d34060-istio-envoy" pod "6dffba47-88ed-4357-862e-d5c6e1d34060" (UID: "6dffba47-88ed-4357-862e-d5c6e1d34060"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:40:37.008104 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.008017    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6dffba47-88ed-4357-862e-d5c6e1d34060-istio-podinfo" pod "6dffba47-88ed-4357-862e-d5c6e1d34060" (UID: "6dffba47-88ed-4357-862e-d5c6e1d34060"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 22:40:37.009790 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.009762    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dffba47-88ed-4357-862e-d5c6e1d34060-kube-api-access-qm4cj" pod "6dffba47-88ed-4357-862e-d5c6e1d34060" (UID: "6dffba47-88ed-4357-862e-d5c6e1d34060"). InnerVolumeSpecName "kube-api-access-qm4cj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:40:37.009887 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.009799    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dffba47-88ed-4357-862e-d5c6e1d34060-istio-token" pod "6dffba47-88ed-4357-862e-d5c6e1d34060" (UID: "6dffba47-88ed-4357-862e-d5c6e1d34060"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:40:37.109984 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.109504    4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dffba47-88ed-4357-862e-d5c6e1d34060-istio-envoy\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:40:37.109984 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.109535    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qm4cj\" (UniqueName: \"kubernetes.io/projected/6dffba47-88ed-4357-862e-d5c6e1d34060-kube-api-access-qm4cj\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:40:37.109984 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.109548    4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dffba47-88ed-4357-862e-d5c6e1d34060-istio-token\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:40:37.109984 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.109562    4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dffba47-88ed-4357-862e-d5c6e1d34060-istio-podinfo\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:40:37.137939 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.137913    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:40:37.287398 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.287368    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf"
Nov 03 22:40:37.407602 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.407521    4431 generic.go:356] "Generic (PLEG): container finished" podID="6dffba47-88ed-4357-862e-d5c6e1d34060" containerID="68a696bb0f8bec8e92449bc8c6ac53a1e84dcbd6fb8657cf4009a1df8f49428b" exitCode=0
Nov 03 22:40:37.407602 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.407574    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-85x5q"
Nov 03 22:40:37.407806 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.407606    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-85x5q" event={"ID":"6dffba47-88ed-4357-862e-d5c6e1d34060","Type":"ContainerDied","Data":"68a696bb0f8bec8e92449bc8c6ac53a1e84dcbd6fb8657cf4009a1df8f49428b"}
Nov 03 22:40:37.407806 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.407662    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-85x5q" event={"ID":"6dffba47-88ed-4357-862e-d5c6e1d34060","Type":"ContainerDied","Data":"8a03c11507dc25a6b310cc9686e9a181193b1b97b4e49c8d1a50e3252b6133bb"}
Nov 03 22:40:37.407806 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.407680    4431 scope.go:122] "RemoveContainer" containerID="68a696bb0f8bec8e92449bc8c6ac53a1e84dcbd6fb8657cf4009a1df8f49428b"
Nov 03 22:40:37.409109 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.409081    4431 generic.go:356] "Generic (PLEG): container finished" podID="4db918b5-a69e-45f7-996a-2658132cbffc" containerID="3e2220b8c700b90cf1c7eff5571b93b90a3c2f6031f4c061ca140100387d0e73" exitCode=0
Nov 03 22:40:37.409188 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.409146    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf"
Nov 03 22:40:37.409188 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.409171    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf" event={"ID":"4db918b5-a69e-45f7-996a-2658132cbffc","Type":"ContainerDied","Data":"3e2220b8c700b90cf1c7eff5571b93b90a3c2f6031f4c061ca140100387d0e73"}
Nov 03 22:40:37.409318 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.409255    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf" event={"ID":"4db918b5-a69e-45f7-996a-2658132cbffc","Type":"ContainerDied","Data":"e2d1d2e2da42f58fc773edb02e6becb6bfb85016f04c7e8e9aebf71f55864cc1"}
Nov 03 22:40:37.410876 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.410851    4431 generic.go:356] "Generic (PLEG): container finished" podID="9a625b54-3dd0-4a42-9583-fe46b592c1a6" containerID="fbcd896eba36a903dbcf4a1e2284b12c4a6807dd92b7b707f876eadafa858038" exitCode=0
Nov 03 22:40:37.411020 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.410896    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9fdlh"
Nov 03 22:40:37.411020 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.410901    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9fdlh" event={"ID":"9a625b54-3dd0-4a42-9583-fe46b592c1a6","Type":"ContainerDied","Data":"fbcd896eba36a903dbcf4a1e2284b12c4a6807dd92b7b707f876eadafa858038"}
Nov 03 22:40:37.411020 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.410937    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9fdlh" event={"ID":"9a625b54-3dd0-4a42-9583-fe46b592c1a6","Type":"ContainerDied","Data":"34f77a7a8b6ad645a198692dbf4195b5193eab14d7325eb4a0c250c018449570"}
Nov 03 22:40:37.413759 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.413722    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4db918b5-a69e-45f7-996a-2658132cbffc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4db918b5-a69e-45f7-996a-2658132cbffc-istiod-ca-cert\") pod \"4db918b5-a69e-45f7-996a-2658132cbffc\" (UID: \"4db918b5-a69e-45f7-996a-2658132cbffc\") "
Nov 03 22:40:37.413890 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.413769    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4db918b5-a69e-45f7-996a-2658132cbffc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4db918b5-a69e-45f7-996a-2658132cbffc-istio-data\") pod \"4db918b5-a69e-45f7-996a-2658132cbffc\" (UID: \"4db918b5-a69e-45f7-996a-2658132cbffc\") "
Nov 03 22:40:37.413890 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.413808    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4db918b5-a69e-45f7-996a-2658132cbffc-istio-token\" (UniqueName: \"kubernetes.io/projected/4db918b5-a69e-45f7-996a-2658132cbffc-istio-token\") pod \"4db918b5-a69e-45f7-996a-2658132cbffc\" (UID: \"4db918b5-a69e-45f7-996a-2658132cbffc\") "
Nov 03 22:40:37.413986 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.413910    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4db918b5-a69e-45f7-996a-2658132cbffc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4db918b5-a69e-45f7-996a-2658132cbffc-istio-podinfo\") pod \"4db918b5-a69e-45f7-996a-2658132cbffc\" (UID: \"4db918b5-a69e-45f7-996a-2658132cbffc\") "
Nov 03 22:40:37.414047 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.414028    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4db918b5-a69e-45f7-996a-2658132cbffc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4db918b5-a69e-45f7-996a-2658132cbffc-workload-socket\") pod \"4db918b5-a69e-45f7-996a-2658132cbffc\" (UID: \"4db918b5-a69e-45f7-996a-2658132cbffc\") "
Nov 03 22:40:37.414086 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.414059    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4db918b5-a69e-45f7-996a-2658132cbffc-istiod-ca-cert" pod "4db918b5-a69e-45f7-996a-2658132cbffc" (UID: "4db918b5-a69e-45f7-996a-2658132cbffc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:40:37.414086 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.414074    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4db918b5-a69e-45f7-996a-2658132cbffc-istio-data" pod "4db918b5-a69e-45f7-996a-2658132cbffc" (UID: "4db918b5-a69e-45f7-996a-2658132cbffc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:40:37.414160 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.414096    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4db918b5-a69e-45f7-996a-2658132cbffc-kube-api-access-zw7h5\" (UniqueName: \"kubernetes.io/projected/4db918b5-a69e-45f7-996a-2658132cbffc-kube-api-access-zw7h5\") pod \"4db918b5-a69e-45f7-996a-2658132cbffc\" (UID: \"4db918b5-a69e-45f7-996a-2658132cbffc\") "
Nov 03 22:40:37.414258 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.414243    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4db918b5-a69e-45f7-996a-2658132cbffc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4db918b5-a69e-45f7-996a-2658132cbffc-istio-envoy\") pod \"4db918b5-a69e-45f7-996a-2658132cbffc\" (UID: \"4db918b5-a69e-45f7-996a-2658132cbffc\") "
Nov 03 22:40:37.414311 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.414284    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4db918b5-a69e-45f7-996a-2658132cbffc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4db918b5-a69e-45f7-996a-2658132cbffc-credential-socket\") pod \"4db918b5-a69e-45f7-996a-2658132cbffc\" (UID: \"4db918b5-a69e-45f7-996a-2658132cbffc\") "
Nov 03 22:40:37.414368 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.414354    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4db918b5-a69e-45f7-996a-2658132cbffc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4db918b5-a69e-45f7-996a-2658132cbffc-workload-certs\") pod \"4db918b5-a69e-45f7-996a-2658132cbffc\" (UID: \"4db918b5-a69e-45f7-996a-2658132cbffc\") "
Nov 03 22:40:37.414756 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.414731    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4db918b5-a69e-45f7-996a-2658132cbffc-workload-socket" pod "4db918b5-a69e-45f7-996a-2658132cbffc" (UID: "4db918b5-a69e-45f7-996a-2658132cbffc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:40:37.414951 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.414932    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4db918b5-a69e-45f7-996a-2658132cbffc-credential-socket" pod "4db918b5-a69e-45f7-996a-2658132cbffc" (UID: "4db918b5-a69e-45f7-996a-2658132cbffc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:40:37.415049 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.415029    4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4db918b5-a69e-45f7-996a-2658132cbffc-workload-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:40:37.415116 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.415053    4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4db918b5-a69e-45f7-996a-2658132cbffc-credential-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:40:37.415116 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.415067    4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4db918b5-a69e-45f7-996a-2658132cbffc-istiod-ca-cert\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:40:37.415116 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.415080    4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4db918b5-a69e-45f7-996a-2658132cbffc-istio-data\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:40:37.415331 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.415312    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4db918b5-a69e-45f7-996a-2658132cbffc-workload-certs" pod "4db918b5-a69e-45f7-996a-2658132cbffc" (UID: "4db918b5-a69e-45f7-996a-2658132cbffc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:40:37.415423 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.415403    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:40:37.416192 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.416171    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4db918b5-a69e-45f7-996a-2658132cbffc-istio-podinfo" pod "4db918b5-a69e-45f7-996a-2658132cbffc" (UID: "4db918b5-a69e-45f7-996a-2658132cbffc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 22:40:37.417173 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.417148    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4db918b5-a69e-45f7-996a-2658132cbffc-istio-token" pod "4db918b5-a69e-45f7-996a-2658132cbffc" (UID: "4db918b5-a69e-45f7-996a-2658132cbffc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:40:37.417506 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.417471    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4db918b5-a69e-45f7-996a-2658132cbffc-istio-envoy" pod "4db918b5-a69e-45f7-996a-2658132cbffc" (UID: "4db918b5-a69e-45f7-996a-2658132cbffc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:40:37.418924 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.418894    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4db918b5-a69e-45f7-996a-2658132cbffc-kube-api-access-zw7h5" pod "4db918b5-a69e-45f7-996a-2658132cbffc" (UID: "4db918b5-a69e-45f7-996a-2658132cbffc"). InnerVolumeSpecName "kube-api-access-zw7h5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:40:37.419820 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.419800    4431 scope.go:122] "RemoveContainer" containerID="68a696bb0f8bec8e92449bc8c6ac53a1e84dcbd6fb8657cf4009a1df8f49428b"
Nov 03 22:40:37.420117 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:40:37.420095    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68a696bb0f8bec8e92449bc8c6ac53a1e84dcbd6fb8657cf4009a1df8f49428b\": container with ID starting with 68a696bb0f8bec8e92449bc8c6ac53a1e84dcbd6fb8657cf4009a1df8f49428b not found: ID does not exist" containerID="68a696bb0f8bec8e92449bc8c6ac53a1e84dcbd6fb8657cf4009a1df8f49428b"
Nov 03 22:40:37.420171 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.420125    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68a696bb0f8bec8e92449bc8c6ac53a1e84dcbd6fb8657cf4009a1df8f49428b"} err="failed to get container status \"68a696bb0f8bec8e92449bc8c6ac53a1e84dcbd6fb8657cf4009a1df8f49428b\": rpc error: code = NotFound desc = could not find container \"68a696bb0f8bec8e92449bc8c6ac53a1e84dcbd6fb8657cf4009a1df8f49428b\": container with ID starting with 68a696bb0f8bec8e92449bc8c6ac53a1e84dcbd6fb8657cf4009a1df8f49428b not found: ID does not exist"
Nov 03 22:40:37.420171 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.420144    4431 scope.go:122] "RemoveContainer" containerID="3e2220b8c700b90cf1c7eff5571b93b90a3c2f6031f4c061ca140100387d0e73"
Nov 03 22:40:37.430163 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.430142    4431 scope.go:122] "RemoveContainer" containerID="3e2220b8c700b90cf1c7eff5571b93b90a3c2f6031f4c061ca140100387d0e73"
Nov 03 22:40:37.430437 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:40:37.430416    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e2220b8c700b90cf1c7eff5571b93b90a3c2f6031f4c061ca140100387d0e73\": container with ID starting with 3e2220b8c700b90cf1c7eff5571b93b90a3c2f6031f4c061ca140100387d0e73 not found: ID does not exist" containerID="3e2220b8c700b90cf1c7eff5571b93b90a3c2f6031f4c061ca140100387d0e73"
Nov 03 22:40:37.430517 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.430447    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e2220b8c700b90cf1c7eff5571b93b90a3c2f6031f4c061ca140100387d0e73"} err="failed to get container status \"3e2220b8c700b90cf1c7eff5571b93b90a3c2f6031f4c061ca140100387d0e73\": rpc error: code = NotFound desc = could not find container \"3e2220b8c700b90cf1c7eff5571b93b90a3c2f6031f4c061ca140100387d0e73\": container with ID starting with 3e2220b8c700b90cf1c7eff5571b93b90a3c2f6031f4c061ca140100387d0e73 not found: ID does not exist"
Nov 03 22:40:37.430517 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.430469    4431 scope.go:122] "RemoveContainer" containerID="fbcd896eba36a903dbcf4a1e2284b12c4a6807dd92b7b707f876eadafa858038"
Nov 03 22:40:37.439944 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.439929    4431 scope.go:122] "RemoveContainer" containerID="fbcd896eba36a903dbcf4a1e2284b12c4a6807dd92b7b707f876eadafa858038"
Nov 03 22:40:37.440237 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:40:37.440199    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbcd896eba36a903dbcf4a1e2284b12c4a6807dd92b7b707f876eadafa858038\": container with ID starting with fbcd896eba36a903dbcf4a1e2284b12c4a6807dd92b7b707f876eadafa858038 not found: ID does not exist" containerID="fbcd896eba36a903dbcf4a1e2284b12c4a6807dd92b7b707f876eadafa858038"
Nov 03 22:40:37.440300 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.440238    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbcd896eba36a903dbcf4a1e2284b12c4a6807dd92b7b707f876eadafa858038"} err="failed to get container status \"fbcd896eba36a903dbcf4a1e2284b12c4a6807dd92b7b707f876eadafa858038\": rpc error: code = NotFound desc = could not find container \"fbcd896eba36a903dbcf4a1e2284b12c4a6807dd92b7b707f876eadafa858038\": container with ID starting with fbcd896eba36a903dbcf4a1e2284b12c4a6807dd92b7b707f876eadafa858038 not found: ID does not exist"
Nov 03 22:40:37.481352 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.481309    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-85x5q"]
Nov 03 22:40:37.482985 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.482915    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-85x5q"]
Nov 03 22:40:37.516674 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.516642    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zw7h5\" (UniqueName: \"kubernetes.io/projected/4db918b5-a69e-45f7-996a-2658132cbffc-kube-api-access-zw7h5\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:40:37.516674 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.516673    4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4db918b5-a69e-45f7-996a-2658132cbffc-istio-envoy\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:40:37.516870 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.516686    4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4db918b5-a69e-45f7-996a-2658132cbffc-workload-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:40:37.516870 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.516699    4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4db918b5-a69e-45f7-996a-2658132cbffc-istio-token\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:40:37.516870 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.516710    4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4db918b5-a69e-45f7-996a-2658132cbffc-istio-podinfo\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:40:37.524757 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.524731    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9fdlh"]
Nov 03 22:40:37.529632 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.529607    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9fdlh"]
Nov 03 22:40:37.713094 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.713043    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:40:37.767076 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.767037    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf"]
Nov 03 22:40:37.771055 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:37.771026    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-thldf"]
Nov 03 22:40:38.088505 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:38.088430    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4db918b5-a69e-45f7-996a-2658132cbffc" path="/var/lib/kubelet/pods/4db918b5-a69e-45f7-996a-2658132cbffc/volumes"
Nov 03 22:40:38.089138 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:38.089115    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6dffba47-88ed-4357-862e-d5c6e1d34060" path="/var/lib/kubelet/pods/6dffba47-88ed-4357-862e-d5c6e1d34060/volumes"
Nov 03 22:40:38.089688 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:38.089673    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a625b54-3dd0-4a42-9583-fe46b592c1a6" path="/var/lib/kubelet/pods/9a625b54-3dd0-4a42-9583-fe46b592c1a6/volumes"
Nov 03 22:40:38.162165 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:38.162131    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:40:38.755238 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:38.755172    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:40:39.019906 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:39.019816    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:40:39.179070 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:39.179036    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:40:40.609299 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:40.609202    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:40:42.423774 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:42.423737    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:40:42.635024 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:42.634984    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:40:44.107706 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:44.107673    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:40:44.185954 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:44.185908    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-71a7df-5f4b644b-zwb9k"]
Nov 03 22:40:44.186577 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:44.186563    4431 cpu_manager.go:414] "containerMap: removing container" podUID="9a625b54-3dd0-4a42-9583-fe46b592c1a6" containerName="istio-proxy"
Nov 03 22:40:44.186630 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:44.186579    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a625b54-3dd0-4a42-9583-fe46b592c1a6" containerName="istio-proxy"
Nov 03 22:40:44.186630 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:44.186601    4431 cpu_manager.go:414] "containerMap: removing container" podUID="6dffba47-88ed-4357-862e-d5c6e1d34060" containerName="istio-proxy"
Nov 03 22:40:44.186630 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:44.186608    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dffba47-88ed-4357-862e-d5c6e1d34060" containerName="istio-proxy"
Nov 03 22:40:44.186765 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:44.186651    4431 cpu_manager.go:414] "containerMap: removing container" podUID="27596c43-8204-4fd0-9ab0-612997367c40" containerName="istio-proxy"
Nov 03 22:40:44.186765 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:44.186658    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27596c43-8204-4fd0-9ab0-612997367c40" containerName="istio-proxy"
Nov 03 22:40:44.186765 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:44.186672    4431 cpu_manager.go:414] "containerMap: removing container" podUID="4db918b5-a69e-45f7-996a-2658132cbffc" containerName="istio-proxy"
Nov 03 22:40:44.186765 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:44.186678    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4db918b5-a69e-45f7-996a-2658132cbffc" containerName="istio-proxy"
Nov 03 22:40:44.186881 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:44.186775    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dffba47-88ed-4357-862e-d5c6e1d34060" containerName="istio-proxy"
Nov 03 22:40:44.186881 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:44.186789    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="27596c43-8204-4fd0-9ab0-612997367c40" containerName="istio-proxy"
Nov 03 22:40:44.186881 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:44.186797    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="4db918b5-a69e-45f7-996a-2658132cbffc" containerName="istio-proxy"
Nov 03 22:40:44.186881 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:44.186805    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a625b54-3dd0-4a42-9583-fe46b592c1a6" containerName="istio-proxy"
Nov 03 22:40:44.191975 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:44.191960    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-71a7df-5f4b644b-zwb9k"
Nov 03 22:40:44.213290 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:44.213261    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-71a7df-5f4b644b-zwb9k"]
Nov 03 22:40:44.217601 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:44.217572    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcbht\" (UniqueName: \"kubernetes.io/projected/53276d80-69e4-482f-a285-370be912c076-kube-api-access-hcbht\") pod \"payload-processing-e2e-inf-b-71a7df-5f4b644b-zwb9k\" (UID: \"53276d80-69e4-482f-a285-370be912c076\") " pod="openshift-ingress/payload-processing-e2e-inf-b-71a7df-5f4b644b-zwb9k"
Nov 03 22:40:44.217795 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:44.217696    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53276d80-69e4-482f-a285-370be912c076-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-71a7df-5f4b644b-zwb9k\" (UID: \"53276d80-69e4-482f-a285-370be912c076\") " pod="openshift-ingress/payload-processing-e2e-inf-b-71a7df-5f4b644b-zwb9k"
Nov 03 22:40:44.220185 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:44.220166    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-71a7df-dockercfg-ktvgs\""
Nov 03 22:40:44.220519 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:44.220492    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-71a7df\""
Nov 03 22:40:44.242262 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:44.242229    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-71a7df-fdd88b4b7-sdbdj"]
Nov 03 22:40:44.247685 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:44.247660    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-71a7df-fdd88b4b7-sdbdj"
Nov 03 22:40:44.265562 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:44.265532    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-71a7df-fdd88b4b7-sdbdj"]
Nov 03 22:40:44.266784 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:44.266761    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:40:44.319437 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:44.319407    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7svv\" (UniqueName: \"kubernetes.io/projected/693e1c0a-7f7c-4a6b-af46-2aa6e1974da5-kube-api-access-m7svv\") pod \"payload-pre-processing-e2e-inf-b-71a7df-fdd88b4b7-sdbdj\" (UID: \"693e1c0a-7f7c-4a6b-af46-2aa6e1974da5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-71a7df-fdd88b4b7-sdbdj"
Nov 03 22:40:44.319597 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:44.319445    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/693e1c0a-7f7c-4a6b-af46-2aa6e1974da5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-71a7df-fdd88b4b7-sdbdj\" (UID: \"693e1c0a-7f7c-4a6b-af46-2aa6e1974da5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-71a7df-fdd88b4b7-sdbdj"
Nov 03 22:40:44.319597 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:44.319519    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcbht\" (UniqueName: \"kubernetes.io/projected/53276d80-69e4-482f-a285-370be912c076-kube-api-access-hcbht\") pod \"payload-processing-e2e-inf-b-71a7df-5f4b644b-zwb9k\" (UID: \"53276d80-69e4-482f-a285-370be912c076\") " pod="openshift-ingress/payload-processing-e2e-inf-b-71a7df-5f4b644b-zwb9k"
Nov 03 22:40:44.319597 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:44.319587    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53276d80-69e4-482f-a285-370be912c076-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-71a7df-5f4b644b-zwb9k\" (UID: \"53276d80-69e4-482f-a285-370be912c076\") " pod="openshift-ingress/payload-processing-e2e-inf-b-71a7df-5f4b644b-zwb9k"
Nov 03 22:40:44.320270 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:44.320250    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53276d80-69e4-482f-a285-370be912c076-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-71a7df-5f4b644b-zwb9k\" (UID: \"53276d80-69e4-482f-a285-370be912c076\") " pod="openshift-ingress/payload-processing-e2e-inf-b-71a7df-5f4b644b-zwb9k"
Nov 03 22:40:44.342462 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:44.342436    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcbht\" (UniqueName: \"kubernetes.io/projected/53276d80-69e4-482f-a285-370be912c076-kube-api-access-hcbht\") pod \"payload-processing-e2e-inf-b-71a7df-5f4b644b-zwb9k\" (UID: \"53276d80-69e4-482f-a285-370be912c076\") " pod="openshift-ingress/payload-processing-e2e-inf-b-71a7df-5f4b644b-zwb9k"
Nov 03 22:40:44.438612 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:44.436910    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7svv\" (UniqueName: \"kubernetes.io/projected/693e1c0a-7f7c-4a6b-af46-2aa6e1974da5-kube-api-access-m7svv\") pod \"payload-pre-processing-e2e-inf-b-71a7df-fdd88b4b7-sdbdj\" (UID: \"693e1c0a-7f7c-4a6b-af46-2aa6e1974da5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-71a7df-fdd88b4b7-sdbdj"
Nov 03 22:40:44.438612 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:44.436979    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/693e1c0a-7f7c-4a6b-af46-2aa6e1974da5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-71a7df-fdd88b4b7-sdbdj\" (UID: \"693e1c0a-7f7c-4a6b-af46-2aa6e1974da5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-71a7df-fdd88b4b7-sdbdj"
Nov 03 22:40:44.438908 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:44.438835    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/693e1c0a-7f7c-4a6b-af46-2aa6e1974da5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-71a7df-fdd88b4b7-sdbdj\" (UID: \"693e1c0a-7f7c-4a6b-af46-2aa6e1974da5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-71a7df-fdd88b4b7-sdbdj"
Nov 03 22:40:44.439604 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:44.439573    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:40:44.491240 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:44.485883    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7svv\" (UniqueName: \"kubernetes.io/projected/693e1c0a-7f7c-4a6b-af46-2aa6e1974da5-kube-api-access-m7svv\") pod \"payload-pre-processing-e2e-inf-b-71a7df-fdd88b4b7-sdbdj\" (UID: \"693e1c0a-7f7c-4a6b-af46-2aa6e1974da5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-71a7df-fdd88b4b7-sdbdj"
Nov 03 22:40:44.511716 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:44.511679    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-71a7df-5f4b644b-zwb9k"
Nov 03 22:40:44.568843 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:44.568800    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-71a7df-fdd88b4b7-sdbdj"
Nov 03 22:40:44.869604 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:44.869575    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-71a7df-5f4b644b-zwb9k"]
Nov 03 22:40:44.870138 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:40:44.870112    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53276d80_69e4_482f_a285_370be912c076.slice/crio-97d7439eca4bb511fcb8d8cb850046fd4883e874aa573ed96c8551e9536c8fd6 WatchSource:0}: Error finding container 97d7439eca4bb511fcb8d8cb850046fd4883e874aa573ed96c8551e9536c8fd6: Status 404 returned error can't find the container with id 97d7439eca4bb511fcb8d8cb850046fd4883e874aa573ed96c8551e9536c8fd6
Nov 03 22:40:44.884167 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:44.884139    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:40:44.900471 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:44.899558    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-71a7df-fdd88b4b7-sdbdj"]
Nov 03 22:40:44.904970 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:40:44.904942    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod693e1c0a_7f7c_4a6b_af46_2aa6e1974da5.slice/crio-9551b0e802f1fca81633f838ec02b4643a3b2bca223d75aacba51fb6ebd0014f WatchSource:0}: Error finding container 9551b0e802f1fca81633f838ec02b4643a3b2bca223d75aacba51fb6ebd0014f: Status 404 returned error can't find the container with id 9551b0e802f1fca81633f838ec02b4643a3b2bca223d75aacba51fb6ebd0014f
Nov 03 22:40:45.041565 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:45.041527    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:40:45.215542 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:45.215511    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:40:45.503091 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:45.502996    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-71a7df-fdd88b4b7-sdbdj" event={"ID":"693e1c0a-7f7c-4a6b-af46-2aa6e1974da5","Type":"ContainerStarted","Data":"957f028e49907d406209661a4165b9692f36e6b6b3e7a90c580efd9181d7483d"}
Nov 03 22:40:45.503091 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:45.503062    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-71a7df-fdd88b4b7-sdbdj" event={"ID":"693e1c0a-7f7c-4a6b-af46-2aa6e1974da5","Type":"ContainerStarted","Data":"9551b0e802f1fca81633f838ec02b4643a3b2bca223d75aacba51fb6ebd0014f"}
Nov 03 22:40:45.503091 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:45.503087    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-71a7df-fdd88b4b7-sdbdj"
Nov 03 22:40:45.505625 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:45.505593    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-71a7df-5f4b644b-zwb9k" event={"ID":"53276d80-69e4-482f-a285-370be912c076","Type":"ContainerStarted","Data":"f70d1e6d72150391be4dfcb606abcc8a3eda8b5f078841db85309734fbbf1ce5"}
Nov 03 22:40:45.505831 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:45.505644    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-71a7df-5f4b644b-zwb9k" event={"ID":"53276d80-69e4-482f-a285-370be912c076","Type":"ContainerStarted","Data":"97d7439eca4bb511fcb8d8cb850046fd4883e874aa573ed96c8551e9536c8fd6"}
Nov 03 22:40:45.505934 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:45.505861    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-71a7df-5f4b644b-zwb9k"
Nov 03 22:40:45.552920 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:45.552863    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-71a7df-fdd88b4b7-sdbdj" podStartSLOduration=1.5528444609999998 podStartE2EDuration="1.552844461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:40:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:40:45.549404009 +0000 UTC m=+1669.575259873" watchObservedRunningTime="2026-11-03 22:40:45.552844461 +0000 UTC m=+1669.578700324"
Nov 03 22:40:45.600429 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:45.600363    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-71a7df-5f4b644b-zwb9k" podStartSLOduration=1.600346429 podStartE2EDuration="1.600346429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:40:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:40:45.598180531 +0000 UTC m=+1669.624036394" watchObservedRunningTime="2026-11-03 22:40:45.600346429 +0000 UTC m=+1669.626202294"
Nov 03 22:40:46.006653 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:46.006609    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:40:46.202977 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:46.202942    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:40:46.648236 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:46.642843    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:40:47.106675 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:47.106636    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:40:47.472238 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:47.471621    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:40:48.116962 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:48.116921    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:40:49.828009 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:49.827969    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:40:50.229448 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:50.229414    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:40:50.376869 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:50.376827    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:40:50.519322 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:50.519237    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:40:50.693868 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:50.693840    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:40:51.507980 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:51.507941    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:40:52.131655 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:52.131624    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:40:52.230028 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:52.229993    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-50b9b5-7f48f98986-ddh6w"]
Nov 03 22:40:52.234464 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:52.234440    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-50b9b5-7f48f98986-ddh6w"
Nov 03 22:40:52.251621 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:52.251596    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-50b9b5-dockercfg-p4fzx\""
Nov 03 22:40:52.255769 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:52.255742    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-50b9b5-7f48f98986-ddh6w"]
Nov 03 22:40:52.266232 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:52.266195    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-50b9b5\""
Nov 03 22:40:52.297568 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:52.297523    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:40:52.397371 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:52.397268    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f000cab-17f2-4a9f-a463-54983bba8566-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-50b9b5-7f48f98986-ddh6w\" (UID: \"5f000cab-17f2-4a9f-a463-54983bba8566\") " pod="openshift-ingress/payload-processing-e2e-inf-a-50b9b5-7f48f98986-ddh6w"
Nov 03 22:40:52.397824 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:52.397797    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfl6v\" (UniqueName: \"kubernetes.io/projected/5f000cab-17f2-4a9f-a463-54983bba8566-kube-api-access-jfl6v\") pod \"payload-processing-e2e-inf-a-50b9b5-7f48f98986-ddh6w\" (UID: \"5f000cab-17f2-4a9f-a463-54983bba8566\") " pod="openshift-ingress/payload-processing-e2e-inf-a-50b9b5-7f48f98986-ddh6w"
Nov 03 22:40:52.484239 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:52.476060    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:40:52.505239 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:52.504048    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f000cab-17f2-4a9f-a463-54983bba8566-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-50b9b5-7f48f98986-ddh6w\" (UID: \"5f000cab-17f2-4a9f-a463-54983bba8566\") " pod="openshift-ingress/payload-processing-e2e-inf-a-50b9b5-7f48f98986-ddh6w"
Nov 03 22:40:52.505239 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:52.504172    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfl6v\" (UniqueName: \"kubernetes.io/projected/5f000cab-17f2-4a9f-a463-54983bba8566-kube-api-access-jfl6v\") pod \"payload-processing-e2e-inf-a-50b9b5-7f48f98986-ddh6w\" (UID: \"5f000cab-17f2-4a9f-a463-54983bba8566\") " pod="openshift-ingress/payload-processing-e2e-inf-a-50b9b5-7f48f98986-ddh6w"
Nov 03 22:40:52.508306 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:52.505608    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f000cab-17f2-4a9f-a463-54983bba8566-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-50b9b5-7f48f98986-ddh6w\" (UID: \"5f000cab-17f2-4a9f-a463-54983bba8566\") " pod="openshift-ingress/payload-processing-e2e-inf-a-50b9b5-7f48f98986-ddh6w"
Nov 03 22:40:52.527998 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:52.527964    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfl6v\" (UniqueName: \"kubernetes.io/projected/5f000cab-17f2-4a9f-a463-54983bba8566-kube-api-access-jfl6v\") pod \"payload-processing-e2e-inf-a-50b9b5-7f48f98986-ddh6w\" (UID: \"5f000cab-17f2-4a9f-a463-54983bba8566\") " pod="openshift-ingress/payload-processing-e2e-inf-a-50b9b5-7f48f98986-ddh6w"
Nov 03 22:40:52.550504 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:52.550468    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-50b9b5-7f48f98986-ddh6w"
Nov 03 22:40:52.865719 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:52.865695    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-50b9b5-7f48f98986-ddh6w"]
Nov 03 22:40:52.867679 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:40:52.867646    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f000cab_17f2_4a9f_a463_54983bba8566.slice/crio-5bd843a3079c473d410a032efb75b6593e3d7067dd08c5c076890ad71e4e3fcd WatchSource:0}: Error finding container 5bd843a3079c473d410a032efb75b6593e3d7067dd08c5c076890ad71e4e3fcd: Status 404 returned error can't find the container with id 5bd843a3079c473d410a032efb75b6593e3d7067dd08c5c076890ad71e4e3fcd
Nov 03 22:40:53.324330 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:53.324283    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:40:53.586572 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:53.586488    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-50b9b5-7f48f98986-ddh6w" event={"ID":"5f000cab-17f2-4a9f-a463-54983bba8566","Type":"ContainerStarted","Data":"fd1b2dc4dced9aba9d1592f9840982e16c9e7e4f336caf7d7cd01e2e23ad6d07"}
Nov 03 22:40:53.586572 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:53.586540    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-50b9b5-7f48f98986-ddh6w" event={"ID":"5f000cab-17f2-4a9f-a463-54983bba8566","Type":"ContainerStarted","Data":"5bd843a3079c473d410a032efb75b6593e3d7067dd08c5c076890ad71e4e3fcd"}
Nov 03 22:40:53.586978 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:53.586713    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-50b9b5-7f48f98986-ddh6w"
Nov 03 22:40:53.627561 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:53.627514    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-50b9b5-7f48f98986-ddh6w" podStartSLOduration=1.627500062 podStartE2EDuration="1.627500062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:40:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:40:53.625228291 +0000 UTC m=+1677.651084148" watchObservedRunningTime="2026-11-03 22:40:53.627500062 +0000 UTC m=+1677.653355924"
Nov 03 22:40:54.914624 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:54.911768    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:40:56.383483 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:56.383451    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:40:56.534387 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:56.534359    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-71a7df-5f4b644b-zwb9k"
Nov 03 22:40:56.534550 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:56.534491    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-71a7df-fdd88b4b7-sdbdj"
Nov 03 22:40:58.130891 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:58.130841    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:40:59.733847 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:40:59.733808    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:41:01.049094 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:01.049053    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:41:01.214022 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:01.213981    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:41:01.266833 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:41:01.266791    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:41:01.361672 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:01.361543    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:41:01.614456 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:01.614339    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc"]
Nov 03 22:41:01.614677 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:01.614543    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc" podUID="36b9d353-1537-445b-bb03-d2d917a2a208" containerName="istio-proxy" containerID="cri-o://f36a69f53131d660f437b23e66abcb043c7a3e3f24581ccdf660bc3cb05c4789" gracePeriod=30
Nov 03 22:41:01.626566 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:01.626536    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh"]
Nov 03 22:41:01.626755 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:01.626731    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh" podUID="6855011d-1f9e-43be-8557-991f928f379e" containerName="istio-proxy" containerID="cri-o://aebb2ba810ac682ca863c77d594a4b3dc6a610a28c1a1d38a749157c2187840e" gracePeriod=30
Nov 03 22:41:03.033708 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:03.033665    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:41:04.597806 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:04.597773    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-50b9b5-7f48f98986-ddh6w"
Nov 03 22:41:05.022145 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:05.022079    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:41:05.621799 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:05.621754    4431 patch_prober.go:29] interesting pod/e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 22:41:05.622272 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:05.621835    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh" podUID="6855011d-1f9e-43be-8557-991f928f379e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 22:41:06.706940 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:06.706905    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:41:06.971061 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:06.971027    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh"
Nov 03 22:41:07.023286 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:07.023244    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc"
Nov 03 22:41:07.025683 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:07.025653    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6855011d-1f9e-43be-8557-991f928f379e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6855011d-1f9e-43be-8557-991f928f379e-istio-data\") pod \"6855011d-1f9e-43be-8557-991f928f379e\" (UID: \"6855011d-1f9e-43be-8557-991f928f379e\") "
Nov 03 22:41:07.025832 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:07.025762    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6855011d-1f9e-43be-8557-991f928f379e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6855011d-1f9e-43be-8557-991f928f379e-credential-socket\") pod \"6855011d-1f9e-43be-8557-991f928f379e\" (UID: \"6855011d-1f9e-43be-8557-991f928f379e\") "
Nov 03 22:41:07.025832 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:07.025799    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6855011d-1f9e-43be-8557-991f928f379e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6855011d-1f9e-43be-8557-991f928f379e-istio-podinfo\") pod \"6855011d-1f9e-43be-8557-991f928f379e\" (UID: \"6855011d-1f9e-43be-8557-991f928f379e\") "
Nov 03 22:41:07.025940 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:07.025844    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6855011d-1f9e-43be-8557-991f928f379e-kube-api-access-5c5r5\" (UniqueName: \"kubernetes.io/projected/6855011d-1f9e-43be-8557-991f928f379e-kube-api-access-5c5r5\") pod \"6855011d-1f9e-43be-8557-991f928f379e\" (UID: \"6855011d-1f9e-43be-8557-991f928f379e\") "
Nov 03 22:41:07.025940 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:07.025927    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6855011d-1f9e-43be-8557-991f928f379e-istio-token\" (UniqueName: \"kubernetes.io/projected/6855011d-1f9e-43be-8557-991f928f379e-istio-token\") pod \"6855011d-1f9e-43be-8557-991f928f379e\" (UID: \"6855011d-1f9e-43be-8557-991f928f379e\") "
Nov 03 22:41:07.026032 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:07.025926    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6855011d-1f9e-43be-8557-991f928f379e-istio-data" pod "6855011d-1f9e-43be-8557-991f928f379e" (UID: "6855011d-1f9e-43be-8557-991f928f379e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:41:07.026111 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:07.026090    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6855011d-1f9e-43be-8557-991f928f379e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6855011d-1f9e-43be-8557-991f928f379e-istio-envoy\") pod \"6855011d-1f9e-43be-8557-991f928f379e\" (UID: \"6855011d-1f9e-43be-8557-991f928f379e\") "
Nov 03 22:41:07.026166 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:07.026141    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6855011d-1f9e-43be-8557-991f928f379e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6855011d-1f9e-43be-8557-991f928f379e-workload-socket\") pod \"6855011d-1f9e-43be-8557-991f928f379e\" (UID: \"6855011d-1f9e-43be-8557-991f928f379e\") "
Nov 03 22:41:07.027033 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:07.026276    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6855011d-1f9e-43be-8557-991f928f379e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6855011d-1f9e-43be-8557-991f928f379e-workload-certs\") pod \"6855011d-1f9e-43be-8557-991f928f379e\" (UID: \"6855011d-1f9e-43be-8557-991f928f379e\") "
Nov 03 22:41:07.027033 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:07.026362    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6855011d-1f9e-43be-8557-991f928f379e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6855011d-1f9e-43be-8557-991f928f379e-istiod-ca-cert\") pod \"6855011d-1f9e-43be-8557-991f928f379e\" (UID: \"6855011d-1f9e-43be-8557-991f928f379e\") "
Nov 03 22:41:07.027033 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:07.026659    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6855011d-1f9e-43be-8557-991f928f379e-workload-certs" pod "6855011d-1f9e-43be-8557-991f928f379e" (UID: "6855011d-1f9e-43be-8557-991f928f379e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:41:07.027033 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:07.026932    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6855011d-1f9e-43be-8557-991f928f379e-istiod-ca-cert" pod "6855011d-1f9e-43be-8557-991f928f379e" (UID: "6855011d-1f9e-43be-8557-991f928f379e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:41:07.027033 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:07.026993    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6855011d-1f9e-43be-8557-991f928f379e-workload-socket" pod "6855011d-1f9e-43be-8557-991f928f379e" (UID: "6855011d-1f9e-43be-8557-991f928f379e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:41:07.027386 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:07.027228    4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6855011d-1f9e-43be-8557-991f928f379e-workload-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:41:07.027386 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:07.027247    4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6855011d-1f9e-43be-8557-991f928f379e-workload-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:41:07.027386 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:07.027261    4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6855011d-1f9e-43be-8557-991f928f379e-istiod-ca-cert\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:41:07.027386 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:07.027273    4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6855011d-1f9e-43be-8557-991f928f379e-istio-data\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:41:07.027679 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:07.027592    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6855011d-1f9e-43be-8557-991f928f379e-credential-socket" pod "6855011d-1f9e-43be-8557-991f928f379e" (UID: "6855011d-1f9e-43be-8557-991f928f379e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:41:07.028861 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:07.028832    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6855011d-1f9e-43be-8557-991f928f379e-istio-podinfo" pod "6855011d-1f9e-43be-8557-991f928f379e" (UID: "6855011d-1f9e-43be-8557-991f928f379e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 22:41:07.029519 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:07.029496    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6855011d-1f9e-43be-8557-991f928f379e-istio-token" pod "6855011d-1f9e-43be-8557-991f928f379e" (UID: "6855011d-1f9e-43be-8557-991f928f379e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:41:07.029860 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:07.029829    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6855011d-1f9e-43be-8557-991f928f379e-istio-envoy" pod "6855011d-1f9e-43be-8557-991f928f379e" (UID: "6855011d-1f9e-43be-8557-991f928f379e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:41:07.030420 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:07.030399    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6855011d-1f9e-43be-8557-991f928f379e-kube-api-access-5c5r5" pod "6855011d-1f9e-43be-8557-991f928f379e" (UID: "6855011d-1f9e-43be-8557-991f928f379e"). InnerVolumeSpecName "kube-api-access-5c5r5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:41:07.127781 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:07.127738    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36b9d353-1537-445b-bb03-d2d917a2a208-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36b9d353-1537-445b-bb03-d2d917a2a208-workload-certs\") pod \"36b9d353-1537-445b-bb03-d2d917a2a208\" (UID: \"36b9d353-1537-445b-bb03-d2d917a2a208\") "
Nov 03 22:41:07.127781 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:07.127784    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36b9d353-1537-445b-bb03-d2d917a2a208-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36b9d353-1537-445b-bb03-d2d917a2a208-istiod-ca-cert\") pod \"36b9d353-1537-445b-bb03-d2d917a2a208\" (UID: \"36b9d353-1537-445b-bb03-d2d917a2a208\") "
Nov 03 22:41:07.128103 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:07.127898    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/36b9d353-1537-445b-bb03-d2d917a2a208-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36b9d353-1537-445b-bb03-d2d917a2a208-istio-podinfo\") pod \"36b9d353-1537-445b-bb03-d2d917a2a208\" (UID: \"36b9d353-1537-445b-bb03-d2d917a2a208\") "
Nov 03 22:41:07.128103 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:07.127969    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36b9d353-1537-445b-bb03-d2d917a2a208-kube-api-access-zns5l\" (UniqueName: \"kubernetes.io/projected/36b9d353-1537-445b-bb03-d2d917a2a208-kube-api-access-zns5l\") pod \"36b9d353-1537-445b-bb03-d2d917a2a208\" (UID: \"36b9d353-1537-445b-bb03-d2d917a2a208\") "
Nov 03 22:41:07.128103 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:07.128008    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36b9d353-1537-445b-bb03-d2d917a2a208-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36b9d353-1537-445b-bb03-d2d917a2a208-istio-data\") pod \"36b9d353-1537-445b-bb03-d2d917a2a208\" (UID: \"36b9d353-1537-445b-bb03-d2d917a2a208\") "
Nov 03 22:41:07.128103 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:07.128056    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36b9d353-1537-445b-bb03-d2d917a2a208-workload-certs" pod "36b9d353-1537-445b-bb03-d2d917a2a208" (UID: "36b9d353-1537-445b-bb03-d2d917a2a208"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:41:07.128103 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:07.128080    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36b9d353-1537-445b-bb03-d2d917a2a208-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36b9d353-1537-445b-bb03-d2d917a2a208-workload-socket\") pod \"36b9d353-1537-445b-bb03-d2d917a2a208\" (UID: \"36b9d353-1537-445b-bb03-d2d917a2a208\") "
Nov 03 22:41:07.128454 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:07.128185    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36b9d353-1537-445b-bb03-d2d917a2a208-istio-token\" (UniqueName: \"kubernetes.io/projected/36b9d353-1537-445b-bb03-d2d917a2a208-istio-token\") pod \"36b9d353-1537-445b-bb03-d2d917a2a208\" (UID: \"36b9d353-1537-445b-bb03-d2d917a2a208\") "
Nov 03 22:41:07.128454 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:07.128253    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36b9d353-1537-445b-bb03-d2d917a2a208-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36b9d353-1537-445b-bb03-d2d917a2a208-credential-socket\") pod \"36b9d353-1537-445b-bb03-d2d917a2a208\" (UID: \"36b9d353-1537-445b-bb03-d2d917a2a208\") "
Nov 03 22:41:07.128454 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:07.128277    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36b9d353-1537-445b-bb03-d2d917a2a208-istiod-ca-cert" pod "36b9d353-1537-445b-bb03-d2d917a2a208" (UID: "36b9d353-1537-445b-bb03-d2d917a2a208"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:41:07.128454 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:07.128347    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36b9d353-1537-445b-bb03-d2d917a2a208-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36b9d353-1537-445b-bb03-d2d917a2a208-istio-envoy\") pod \"36b9d353-1537-445b-bb03-d2d917a2a208\" (UID: \"36b9d353-1537-445b-bb03-d2d917a2a208\") "
Nov 03 22:41:07.128741 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:07.128661    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36b9d353-1537-445b-bb03-d2d917a2a208-istio-data" pod "36b9d353-1537-445b-bb03-d2d917a2a208" (UID: "36b9d353-1537-445b-bb03-d2d917a2a208"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:41:07.129163 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:07.129134    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36b9d353-1537-445b-bb03-d2d917a2a208-workload-socket" pod "36b9d353-1537-445b-bb03-d2d917a2a208" (UID: "36b9d353-1537-445b-bb03-d2d917a2a208"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:41:07.129253 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:07.129192    4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6855011d-1f9e-43be-8557-991f928f379e-credential-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:41:07.129253 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:07.129230    4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36b9d353-1537-445b-bb03-d2d917a2a208-workload-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:41:07.129253 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:07.129246    4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6855011d-1f9e-43be-8557-991f928f379e-istio-podinfo\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:41:07.129411 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:07.129260    4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36b9d353-1537-445b-bb03-d2d917a2a208-istiod-ca-cert\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:41:07.129411 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:07.129273    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5c5r5\" (UniqueName: \"kubernetes.io/projected/6855011d-1f9e-43be-8557-991f928f379e-kube-api-access-5c5r5\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:41:07.129411 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:07.129286    4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6855011d-1f9e-43be-8557-991f928f379e-istio-token\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:41:07.129411 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:07.129301    4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36b9d353-1537-445b-bb03-d2d917a2a208-istio-data\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:41:07.129411 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:07.129313    4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6855011d-1f9e-43be-8557-991f928f379e-istio-envoy\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:41:07.129679 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:07.129654    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36b9d353-1537-445b-bb03-d2d917a2a208-credential-socket" pod "36b9d353-1537-445b-bb03-d2d917a2a208" (UID: "36b9d353-1537-445b-bb03-d2d917a2a208"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:41:07.141021 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:07.140944    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36b9d353-1537-445b-bb03-d2d917a2a208-istio-token" pod "36b9d353-1537-445b-bb03-d2d917a2a208" (UID: "36b9d353-1537-445b-bb03-d2d917a2a208"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:41:07.141021 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:07.140956    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36b9d353-1537-445b-bb03-d2d917a2a208-istio-envoy" pod "36b9d353-1537-445b-bb03-d2d917a2a208" (UID: "36b9d353-1537-445b-bb03-d2d917a2a208"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:41:07.141021 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:07.140996    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36b9d353-1537-445b-bb03-d2d917a2a208-kube-api-access-zns5l" pod "36b9d353-1537-445b-bb03-d2d917a2a208" (UID: "36b9d353-1537-445b-bb03-d2d917a2a208"). InnerVolumeSpecName "kube-api-access-zns5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:41:07.142427 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:07.142407    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/36b9d353-1537-445b-bb03-d2d917a2a208-istio-podinfo" pod "36b9d353-1537-445b-bb03-d2d917a2a208" (UID: "36b9d353-1537-445b-bb03-d2d917a2a208"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 22:41:07.231112 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:07.231069    4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36b9d353-1537-445b-bb03-d2d917a2a208-istio-podinfo\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:41:07.231112 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:07.231104    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zns5l\" (UniqueName: \"kubernetes.io/projected/36b9d353-1537-445b-bb03-d2d917a2a208-kube-api-access-zns5l\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:41:07.231355 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:07.231122    4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36b9d353-1537-445b-bb03-d2d917a2a208-workload-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:41:07.231355 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:07.231138    4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36b9d353-1537-445b-bb03-d2d917a2a208-istio-token\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:41:07.231355 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:07.231151    4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36b9d353-1537-445b-bb03-d2d917a2a208-credential-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:41:07.231355 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:07.231166    4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36b9d353-1537-445b-bb03-d2d917a2a208-istio-envoy\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:41:07.690555 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:07.690519    4431 generic.go:356] "Generic (PLEG): container finished" podID="6855011d-1f9e-43be-8557-991f928f379e" containerID="aebb2ba810ac682ca863c77d594a4b3dc6a610a28c1a1d38a749157c2187840e" exitCode=0
Nov 03 22:41:07.690742 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:07.690577    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh"
Nov 03 22:41:07.690742 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:07.690595    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh" event={"ID":"6855011d-1f9e-43be-8557-991f928f379e","Type":"ContainerDied","Data":"aebb2ba810ac682ca863c77d594a4b3dc6a610a28c1a1d38a749157c2187840e"}
Nov 03 22:41:07.690742 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:07.690651    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh" event={"ID":"6855011d-1f9e-43be-8557-991f928f379e","Type":"ContainerDied","Data":"7f1553beb8de03809ddb93d30a7990bdec51366a28caae39f791b8fa48e38fd6"}
Nov 03 22:41:07.690742 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:07.690675    4431 scope.go:122] "RemoveContainer" containerID="aebb2ba810ac682ca863c77d594a4b3dc6a610a28c1a1d38a749157c2187840e"
Nov 03 22:41:07.692228 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:07.692188    4431 generic.go:356] "Generic (PLEG): container finished" podID="36b9d353-1537-445b-bb03-d2d917a2a208" containerID="f36a69f53131d660f437b23e66abcb043c7a3e3f24581ccdf660bc3cb05c4789" exitCode=0
Nov 03 22:41:07.692335 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:07.692255    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc" event={"ID":"36b9d353-1537-445b-bb03-d2d917a2a208","Type":"ContainerDied","Data":"f36a69f53131d660f437b23e66abcb043c7a3e3f24581ccdf660bc3cb05c4789"}
Nov 03 22:41:07.692335 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:07.692312    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc" event={"ID":"36b9d353-1537-445b-bb03-d2d917a2a208","Type":"ContainerDied","Data":"56f6e271ed6e1101bd73f614274476fb460b6ab669483ddefff55b9a623aaf30"}
Nov 03 22:41:07.692335 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:07.692263    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc"
Nov 03 22:41:07.711837 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:07.711737    4431 scope.go:122] "RemoveContainer" containerID="aebb2ba810ac682ca863c77d594a4b3dc6a610a28c1a1d38a749157c2187840e"
Nov 03 22:41:07.714985 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:41:07.714952    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aebb2ba810ac682ca863c77d594a4b3dc6a610a28c1a1d38a749157c2187840e\": container with ID starting with aebb2ba810ac682ca863c77d594a4b3dc6a610a28c1a1d38a749157c2187840e not found: ID does not exist" containerID="aebb2ba810ac682ca863c77d594a4b3dc6a610a28c1a1d38a749157c2187840e"
Nov 03 22:41:07.715114 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:07.714993    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aebb2ba810ac682ca863c77d594a4b3dc6a610a28c1a1d38a749157c2187840e"} err="failed to get container status \"aebb2ba810ac682ca863c77d594a4b3dc6a610a28c1a1d38a749157c2187840e\": rpc error: code = NotFound desc = could not find container \"aebb2ba810ac682ca863c77d594a4b3dc6a610a28c1a1d38a749157c2187840e\": container with ID starting with aebb2ba810ac682ca863c77d594a4b3dc6a610a28c1a1d38a749157c2187840e not found: ID does not exist"
Nov 03 22:41:07.715114 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:07.715019    4431 scope.go:122] "RemoveContainer" containerID="f36a69f53131d660f437b23e66abcb043c7a3e3f24581ccdf660bc3cb05c4789"
Nov 03 22:41:07.724874 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:07.724849    4431 scope.go:122] "RemoveContainer" containerID="f36a69f53131d660f437b23e66abcb043c7a3e3f24581ccdf660bc3cb05c4789"
Nov 03 22:41:07.725170 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:41:07.725143    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f36a69f53131d660f437b23e66abcb043c7a3e3f24581ccdf660bc3cb05c4789\": container with ID starting with f36a69f53131d660f437b23e66abcb043c7a3e3f24581ccdf660bc3cb05c4789 not found: ID does not exist" containerID="f36a69f53131d660f437b23e66abcb043c7a3e3f24581ccdf660bc3cb05c4789"
Nov 03 22:41:07.726079 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:07.725171    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f36a69f53131d660f437b23e66abcb043c7a3e3f24581ccdf660bc3cb05c4789"} err="failed to get container status \"f36a69f53131d660f437b23e66abcb043c7a3e3f24581ccdf660bc3cb05c4789\": rpc error: code = NotFound desc = could not find container \"f36a69f53131d660f437b23e66abcb043c7a3e3f24581ccdf660bc3cb05c4789\": container with ID starting with f36a69f53131d660f437b23e66abcb043c7a3e3f24581ccdf660bc3cb05c4789 not found: ID does not exist"
Nov 03 22:41:07.759667 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:07.759632    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc"]
Nov 03 22:41:07.773305 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:07.773266    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-q88pc"]
Nov 03 22:41:07.803906 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:07.803870    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh"]
Nov 03 22:41:07.819492 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:07.819465    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-qslxh"]
Nov 03 22:41:08.089224 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:08.089182    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36b9d353-1537-445b-bb03-d2d917a2a208" path="/var/lib/kubelet/pods/36b9d353-1537-445b-bb03-d2d917a2a208/volumes"
Nov 03 22:41:08.089774 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:08.089757    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6855011d-1f9e-43be-8557-991f928f379e" path="/var/lib/kubelet/pods/6855011d-1f9e-43be-8557-991f928f379e/volumes"
Nov 03 22:41:08.173589 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:08.173556    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-22edc1-684d7dd455-c6bwb"]
Nov 03 22:41:08.173757 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:08.173737    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22edc1-684d7dd455-c6bwb" podUID="a1475fef-4430-4907-9ddc-41aba6c3562c" containerName="maas-api" containerID="cri-o://a08f6ead02b41cf57ffe2c23c56816520f574187b52d3a21064b6fe40657877c" gracePeriod=30
Nov 03 22:41:08.301048 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:08.301012    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:41:08.581236 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:08.566439    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22edc1-684d7dd455-c6bwb"
Nov 03 22:41:08.672239 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:08.667715    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1475fef-4430-4907-9ddc-41aba6c3562c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1475fef-4430-4907-9ddc-41aba6c3562c-metrics-certs\") pod \"a1475fef-4430-4907-9ddc-41aba6c3562c\" (UID: \"a1475fef-4430-4907-9ddc-41aba6c3562c\") "
Nov 03 22:41:08.672239 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:08.667834    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1475fef-4430-4907-9ddc-41aba6c3562c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1475fef-4430-4907-9ddc-41aba6c3562c-openshift-service-ca\") pod \"a1475fef-4430-4907-9ddc-41aba6c3562c\" (UID: \"a1475fef-4430-4907-9ddc-41aba6c3562c\") "
Nov 03 22:41:08.672239 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:08.667902    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1475fef-4430-4907-9ddc-41aba6c3562c-kube-api-access-mhbr7\" (UniqueName: \"kubernetes.io/projected/a1475fef-4430-4907-9ddc-41aba6c3562c-kube-api-access-mhbr7\") pod \"a1475fef-4430-4907-9ddc-41aba6c3562c\" (UID: \"a1475fef-4430-4907-9ddc-41aba6c3562c\") "
Nov 03 22:41:08.672239 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:08.667998    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1475fef-4430-4907-9ddc-41aba6c3562c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1475fef-4430-4907-9ddc-41aba6c3562c-maas-api-tls\") pod \"a1475fef-4430-4907-9ddc-41aba6c3562c\" (UID: \"a1475fef-4430-4907-9ddc-41aba6c3562c\") "
Nov 03 22:41:08.705813 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:08.697315    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1475fef-4430-4907-9ddc-41aba6c3562c-maas-api-tls" pod "a1475fef-4430-4907-9ddc-41aba6c3562c" (UID: "a1475fef-4430-4907-9ddc-41aba6c3562c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:41:08.705813 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:08.704623    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1475fef-4430-4907-9ddc-41aba6c3562c-metrics-certs" pod "a1475fef-4430-4907-9ddc-41aba6c3562c" (UID: "a1475fef-4430-4907-9ddc-41aba6c3562c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:41:08.705813 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:08.704802    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1475fef-4430-4907-9ddc-41aba6c3562c-openshift-service-ca" pod "a1475fef-4430-4907-9ddc-41aba6c3562c" (UID: "a1475fef-4430-4907-9ddc-41aba6c3562c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:41:08.735547 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:08.735510    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1475fef-4430-4907-9ddc-41aba6c3562c-kube-api-access-mhbr7" pod "a1475fef-4430-4907-9ddc-41aba6c3562c" (UID: "a1475fef-4430-4907-9ddc-41aba6c3562c"). InnerVolumeSpecName "kube-api-access-mhbr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:41:08.790019 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:08.788939    4431 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1475fef-4430-4907-9ddc-41aba6c3562c-metrics-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:41:08.790019 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:08.788976    4431 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1475fef-4430-4907-9ddc-41aba6c3562c-openshift-service-ca\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:41:08.790019 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:08.788991    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhbr7\" (UniqueName: \"kubernetes.io/projected/a1475fef-4430-4907-9ddc-41aba6c3562c-kube-api-access-mhbr7\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:41:08.790019 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:08.789005    4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1475fef-4430-4907-9ddc-41aba6c3562c-maas-api-tls\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:41:08.842547 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:08.842512    4431 generic.go:356] "Generic (PLEG): container finished" podID="a1475fef-4430-4907-9ddc-41aba6c3562c" containerID="a08f6ead02b41cf57ffe2c23c56816520f574187b52d3a21064b6fe40657877c" exitCode=0
Nov 03 22:41:08.842733 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:08.842604    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22edc1-684d7dd455-c6bwb" event={"ID":"a1475fef-4430-4907-9ddc-41aba6c3562c","Type":"ContainerDied","Data":"a08f6ead02b41cf57ffe2c23c56816520f574187b52d3a21064b6fe40657877c"}
Nov 03 22:41:08.842733 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:08.842653    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22edc1-684d7dd455-c6bwb" event={"ID":"a1475fef-4430-4907-9ddc-41aba6c3562c","Type":"ContainerDied","Data":"46131b0ae91a4389beadca32c01f896b84e7f65d26a9f376241c43b5b3d5d9e2"}
Nov 03 22:41:08.842733 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:08.842681    4431 scope.go:122] "RemoveContainer" containerID="a08f6ead02b41cf57ffe2c23c56816520f574187b52d3a21064b6fe40657877c"
Nov 03 22:41:08.842913 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:08.842798    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22edc1-684d7dd455-c6bwb"
Nov 03 22:41:08.860692 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:08.860663    4431 scope.go:122] "RemoveContainer" containerID="a08f6ead02b41cf57ffe2c23c56816520f574187b52d3a21064b6fe40657877c"
Nov 03 22:41:08.861113 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:41:08.861087    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a08f6ead02b41cf57ffe2c23c56816520f574187b52d3a21064b6fe40657877c\": container with ID starting with a08f6ead02b41cf57ffe2c23c56816520f574187b52d3a21064b6fe40657877c not found: ID does not exist" containerID="a08f6ead02b41cf57ffe2c23c56816520f574187b52d3a21064b6fe40657877c"
Nov 03 22:41:08.861204 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:08.861127    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a08f6ead02b41cf57ffe2c23c56816520f574187b52d3a21064b6fe40657877c"} err="failed to get container status \"a08f6ead02b41cf57ffe2c23c56816520f574187b52d3a21064b6fe40657877c\": rpc error: code = NotFound desc = could not find container \"a08f6ead02b41cf57ffe2c23c56816520f574187b52d3a21064b6fe40657877c\": container with ID starting with a08f6ead02b41cf57ffe2c23c56816520f574187b52d3a21064b6fe40657877c not found: ID does not exist"
Nov 03 22:41:08.895440 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:08.895408    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-22edc1-684d7dd455-c6bwb"]
Nov 03 22:41:08.899397 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:08.899370    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-22edc1-684d7dd455-c6bwb"]
Nov 03 22:41:09.246608 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:09.246575    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:41:09.546156 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:09.546074    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:41:10.033991 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:10.033962    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:41:10.088885 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:10.088855    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1475fef-4430-4907-9ddc-41aba6c3562c" path="/var/lib/kubelet/pods/a1475fef-4430-4907-9ddc-41aba6c3562c/volumes"
Nov 03 22:41:11.173625 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:11.173592    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7940ea-565f5768b6-8nzcp"]
Nov 03 22:41:11.174284 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:11.174268    4431 cpu_manager.go:414] "containerMap: removing container" podUID="a1475fef-4430-4907-9ddc-41aba6c3562c" containerName="maas-api"
Nov 03 22:41:11.174339 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:11.174285    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1475fef-4430-4907-9ddc-41aba6c3562c" containerName="maas-api"
Nov 03 22:41:11.174339 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:11.174323    4431 cpu_manager.go:414] "containerMap: removing container" podUID="36b9d353-1537-445b-bb03-d2d917a2a208" containerName="istio-proxy"
Nov 03 22:41:11.174339 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:11.174329    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36b9d353-1537-445b-bb03-d2d917a2a208" containerName="istio-proxy"
Nov 03 22:41:11.174433 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:11.174362    4431 cpu_manager.go:414] "containerMap: removing container" podUID="6855011d-1f9e-43be-8557-991f928f379e" containerName="istio-proxy"
Nov 03 22:41:11.174433 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:11.174367    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6855011d-1f9e-43be-8557-991f928f379e" containerName="istio-proxy"
Nov 03 22:41:11.174492 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:11.174459    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="6855011d-1f9e-43be-8557-991f928f379e" containerName="istio-proxy"
Nov 03 22:41:11.174492 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:11.174473    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1475fef-4430-4907-9ddc-41aba6c3562c" containerName="maas-api"
Nov 03 22:41:11.174492 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:11.174484    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="36b9d353-1537-445b-bb03-d2d917a2a208" containerName="istio-proxy"
Nov 03 22:41:11.183744 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:11.183716    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7940ea-565f5768b6-8nzcp"
Nov 03 22:41:11.200076 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:11.200045    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7940ea-565f5768b6-8nzcp"]
Nov 03 22:41:11.207360 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:11.207328    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-7940ea-serving-cert\""
Nov 03 22:41:11.299343 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:11.299303    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:41:11.324912 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:11.324877    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb6vk\" (UniqueName: \"kubernetes.io/projected/789373aa-d138-4dff-a505-17410f8cf0c7-kube-api-access-sb6vk\") pod \"maas-api-e2e-inf-a-7940ea-565f5768b6-8nzcp\" (UID: \"789373aa-d138-4dff-a505-17410f8cf0c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7940ea-565f5768b6-8nzcp"
Nov 03 22:41:11.325084 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:11.324943    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/789373aa-d138-4dff-a505-17410f8cf0c7-metrics-certs\") pod \"maas-api-e2e-inf-a-7940ea-565f5768b6-8nzcp\" (UID: \"789373aa-d138-4dff-a505-17410f8cf0c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7940ea-565f5768b6-8nzcp"
Nov 03 22:41:11.325084 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:11.325076    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/789373aa-d138-4dff-a505-17410f8cf0c7-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7940ea-565f5768b6-8nzcp\" (UID: \"789373aa-d138-4dff-a505-17410f8cf0c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7940ea-565f5768b6-8nzcp"
Nov 03 22:41:11.325205 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:11.325189    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/789373aa-d138-4dff-a505-17410f8cf0c7-maas-api-tls\") pod \"maas-api-e2e-inf-a-7940ea-565f5768b6-8nzcp\" (UID: \"789373aa-d138-4dff-a505-17410f8cf0c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7940ea-565f5768b6-8nzcp"
Nov 03 22:41:11.427537 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:11.427439    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb6vk\" (UniqueName: \"kubernetes.io/projected/789373aa-d138-4dff-a505-17410f8cf0c7-kube-api-access-sb6vk\") pod \"maas-api-e2e-inf-a-7940ea-565f5768b6-8nzcp\" (UID: \"789373aa-d138-4dff-a505-17410f8cf0c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7940ea-565f5768b6-8nzcp"
Nov 03 22:41:11.427702 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:11.427543    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/789373aa-d138-4dff-a505-17410f8cf0c7-metrics-certs\") pod \"maas-api-e2e-inf-a-7940ea-565f5768b6-8nzcp\" (UID: \"789373aa-d138-4dff-a505-17410f8cf0c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7940ea-565f5768b6-8nzcp"
Nov 03 22:41:11.427702 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:11.427637    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/789373aa-d138-4dff-a505-17410f8cf0c7-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7940ea-565f5768b6-8nzcp\" (UID: \"789373aa-d138-4dff-a505-17410f8cf0c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7940ea-565f5768b6-8nzcp"
Nov 03 22:41:11.427832 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:11.427781    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/789373aa-d138-4dff-a505-17410f8cf0c7-maas-api-tls\") pod \"maas-api-e2e-inf-a-7940ea-565f5768b6-8nzcp\" (UID: \"789373aa-d138-4dff-a505-17410f8cf0c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7940ea-565f5768b6-8nzcp"
Nov 03 22:41:11.431376 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:11.431346    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/789373aa-d138-4dff-a505-17410f8cf0c7-maas-api-tls\") pod \"maas-api-e2e-inf-a-7940ea-565f5768b6-8nzcp\" (UID: \"789373aa-d138-4dff-a505-17410f8cf0c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7940ea-565f5768b6-8nzcp"
Nov 03 22:41:11.431999 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:11.431973    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/789373aa-d138-4dff-a505-17410f8cf0c7-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7940ea-565f5768b6-8nzcp\" (UID: \"789373aa-d138-4dff-a505-17410f8cf0c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7940ea-565f5768b6-8nzcp"
Nov 03 22:41:11.435825 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:11.435799    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/789373aa-d138-4dff-a505-17410f8cf0c7-metrics-certs\") pod \"maas-api-e2e-inf-a-7940ea-565f5768b6-8nzcp\" (UID: \"789373aa-d138-4dff-a505-17410f8cf0c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7940ea-565f5768b6-8nzcp"
Nov 03 22:41:11.444536 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:11.444509    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:41:11.457115 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:11.457084    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb6vk\" (UniqueName: \"kubernetes.io/projected/789373aa-d138-4dff-a505-17410f8cf0c7-kube-api-access-sb6vk\") pod \"maas-api-e2e-inf-a-7940ea-565f5768b6-8nzcp\" (UID: \"789373aa-d138-4dff-a505-17410f8cf0c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7940ea-565f5768b6-8nzcp"
Nov 03 22:41:11.501982 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:11.501949    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7940ea-565f5768b6-8nzcp"
Nov 03 22:41:11.593500 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:11.593437    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:41:11.808124 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:11.808095    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7940ea-565f5768b6-8nzcp"]
Nov 03 22:41:11.808349 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:41:11.808326    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod789373aa_d138_4dff_a505_17410f8cf0c7.slice/crio-e4bbf8bc8f2579f8c54adc3558267acbaaf9d9f39bb95c1d5c6f1a02eeae6f8f WatchSource:0}: Error finding container e4bbf8bc8f2579f8c54adc3558267acbaaf9d9f39bb95c1d5c6f1a02eeae6f8f: Status 404 returned error can't find the container with id e4bbf8bc8f2579f8c54adc3558267acbaaf9d9f39bb95c1d5c6f1a02eeae6f8f
Nov 03 22:41:11.809775 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:11.809755    4431 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 22:41:11.867336 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:11.867299    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7940ea-565f5768b6-8nzcp" event={"ID":"789373aa-d138-4dff-a505-17410f8cf0c7","Type":"ContainerStarted","Data":"e4bbf8bc8f2579f8c54adc3558267acbaaf9d9f39bb95c1d5c6f1a02eeae6f8f"}
Nov 03 22:41:12.875427 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:12.875383    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7940ea-565f5768b6-8nzcp" event={"ID":"789373aa-d138-4dff-a505-17410f8cf0c7","Type":"ContainerStarted","Data":"5e85065346e882020a2dca2a640cbad3a08b96ee865fcc60d3c6febf1c129745"}
Nov 03 22:41:12.875842 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:12.875550    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7940ea-565f5768b6-8nzcp"
Nov 03 22:41:12.921114 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:12.921062    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7940ea-565f5768b6-8nzcp" podStartSLOduration=1.628013232 podStartE2EDuration="1.921049004s" totalImagesPullingTime="293.035772ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:41:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:41:12.91796254 +0000 UTC m=+1696.943818403" watchObservedRunningTime="2026-11-03 22:41:12.921049004 +0000 UTC m=+1696.946904866"
Nov 03 22:41:12.961938 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:12.961907    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:41:13.154969 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:13.154851    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:41:14.790202 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:14.790166    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:41:15.769250 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:15.769178    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:41:16.601582 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:16.601539    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:41:18.273862 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:18.273820    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:41:18.422008 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:18.421978    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:41:18.664778 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:18.664749    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:41:18.888849 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:18.888821    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7940ea-565f5768b6-8nzcp"
Nov 03 22:41:19.432911 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:19.432878    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:41:19.985194 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:19.985146    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq"]
Nov 03 22:41:20.002261 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:20.002086    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:41:20.037322 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:20.025517    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq"]
Nov 03 22:41:20.037322 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:20.025650    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq"
Nov 03 22:41:20.064987 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:20.043299    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-bb1fa5-openshift-default-dockercfg-zb9dq\""
Nov 03 22:41:20.119242 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:20.117893    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51131709-120f-4b00-a6ec-34e879ac168b-istio-envoy\") pod \"e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq\" (UID: \"51131709-120f-4b00-a6ec-34e879ac168b\") " pod="openshift-ingress/e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq"
Nov 03 22:41:20.119242 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:20.117946    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51131709-120f-4b00-a6ec-34e879ac168b-istio-podinfo\") pod \"e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq\" (UID: \"51131709-120f-4b00-a6ec-34e879ac168b\") " pod="openshift-ingress/e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq"
Nov 03 22:41:20.119242 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:20.118048    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmt69\" (UniqueName: \"kubernetes.io/projected/51131709-120f-4b00-a6ec-34e879ac168b-kube-api-access-pmt69\") pod \"e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq\" (UID: \"51131709-120f-4b00-a6ec-34e879ac168b\") " pod="openshift-ingress/e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq"
Nov 03 22:41:20.119242 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:20.118137    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51131709-120f-4b00-a6ec-34e879ac168b-credential-socket\") pod \"e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq\" (UID: \"51131709-120f-4b00-a6ec-34e879ac168b\") " pod="openshift-ingress/e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq"
Nov 03 22:41:20.119242 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:20.118172    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51131709-120f-4b00-a6ec-34e879ac168b-istio-token\") pod \"e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq\" (UID: \"51131709-120f-4b00-a6ec-34e879ac168b\") " pod="openshift-ingress/e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq"
Nov 03 22:41:20.119242 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:20.118233    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51131709-120f-4b00-a6ec-34e879ac168b-workload-socket\") pod \"e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq\" (UID: \"51131709-120f-4b00-a6ec-34e879ac168b\") " pod="openshift-ingress/e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq"
Nov 03 22:41:20.119242 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:20.118303    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51131709-120f-4b00-a6ec-34e879ac168b-workload-certs\") pod \"e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq\" (UID: \"51131709-120f-4b00-a6ec-34e879ac168b\") " pod="openshift-ingress/e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq"
Nov 03 22:41:20.119242 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:20.118343    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51131709-120f-4b00-a6ec-34e879ac168b-istio-data\") pod \"e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq\" (UID: \"51131709-120f-4b00-a6ec-34e879ac168b\") " pod="openshift-ingress/e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq"
Nov 03 22:41:20.119242 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:20.118385    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51131709-120f-4b00-a6ec-34e879ac168b-istiod-ca-cert\") pod \"e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq\" (UID: \"51131709-120f-4b00-a6ec-34e879ac168b\") " pod="openshift-ingress/e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq"
Nov 03 22:41:20.219761 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:20.219726    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51131709-120f-4b00-a6ec-34e879ac168b-istio-envoy\") pod \"e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq\" (UID: \"51131709-120f-4b00-a6ec-34e879ac168b\") " pod="openshift-ingress/e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq"
Nov 03 22:41:20.219761 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:20.219768    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51131709-120f-4b00-a6ec-34e879ac168b-istio-podinfo\") pod \"e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq\" (UID: \"51131709-120f-4b00-a6ec-34e879ac168b\") " pod="openshift-ingress/e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq"
Nov 03 22:41:20.220039 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:20.219829    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmt69\" (UniqueName: \"kubernetes.io/projected/51131709-120f-4b00-a6ec-34e879ac168b-kube-api-access-pmt69\") pod \"e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq\" (UID: \"51131709-120f-4b00-a6ec-34e879ac168b\") " pod="openshift-ingress/e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq"
Nov 03 22:41:20.220126 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:20.220032    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51131709-120f-4b00-a6ec-34e879ac168b-credential-socket\") pod \"e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq\" (UID: \"51131709-120f-4b00-a6ec-34e879ac168b\") " pod="openshift-ingress/e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq"
Nov 03 22:41:20.220126 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:20.220074    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51131709-120f-4b00-a6ec-34e879ac168b-istio-token\") pod \"e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq\" (UID: \"51131709-120f-4b00-a6ec-34e879ac168b\") " pod="openshift-ingress/e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq"
Nov 03 22:41:20.220254 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:20.220123    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51131709-120f-4b00-a6ec-34e879ac168b-workload-socket\") pod \"e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq\" (UID: \"51131709-120f-4b00-a6ec-34e879ac168b\") " pod="openshift-ingress/e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq"
Nov 03 22:41:20.220254 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:20.220199    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51131709-120f-4b00-a6ec-34e879ac168b-workload-certs\") pod \"e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq\" (UID: \"51131709-120f-4b00-a6ec-34e879ac168b\") " pod="openshift-ingress/e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq"
Nov 03 22:41:20.220364 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:20.220257    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51131709-120f-4b00-a6ec-34e879ac168b-istio-data\") pod \"e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq\" (UID: \"51131709-120f-4b00-a6ec-34e879ac168b\") " pod="openshift-ingress/e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq"
Nov 03 22:41:20.220427 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:20.220374    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51131709-120f-4b00-a6ec-34e879ac168b-istiod-ca-cert\") pod \"e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq\" (UID: \"51131709-120f-4b00-a6ec-34e879ac168b\") " pod="openshift-ingress/e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq"
Nov 03 22:41:20.220516 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:20.220486    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51131709-120f-4b00-a6ec-34e879ac168b-credential-socket\") pod \"e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq\" (UID: \"51131709-120f-4b00-a6ec-34e879ac168b\") " pod="openshift-ingress/e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq"
Nov 03 22:41:20.220574 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:20.220564    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51131709-120f-4b00-a6ec-34e879ac168b-istio-data\") pod \"e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq\" (UID: \"51131709-120f-4b00-a6ec-34e879ac168b\") " pod="openshift-ingress/e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq"
Nov 03 22:41:20.220815 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:20.220792    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51131709-120f-4b00-a6ec-34e879ac168b-workload-socket\") pod \"e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq\" (UID: \"51131709-120f-4b00-a6ec-34e879ac168b\") " pod="openshift-ingress/e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq"
Nov 03 22:41:20.220890 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:20.220814    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51131709-120f-4b00-a6ec-34e879ac168b-workload-certs\") pod \"e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq\" (UID: \"51131709-120f-4b00-a6ec-34e879ac168b\") " pod="openshift-ingress/e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq"
Nov 03 22:41:20.220984 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:20.220957    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51131709-120f-4b00-a6ec-34e879ac168b-istiod-ca-cert\") pod \"e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq\" (UID: \"51131709-120f-4b00-a6ec-34e879ac168b\") " pod="openshift-ingress/e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq"
Nov 03 22:41:20.222773 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:20.222740    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51131709-120f-4b00-a6ec-34e879ac168b-istio-envoy\") pod \"e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq\" (UID: \"51131709-120f-4b00-a6ec-34e879ac168b\") " pod="openshift-ingress/e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq"
Nov 03 22:41:20.222879 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:20.222851    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51131709-120f-4b00-a6ec-34e879ac168b-istio-podinfo\") pod \"e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq\" (UID: \"51131709-120f-4b00-a6ec-34e879ac168b\") " pod="openshift-ingress/e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq"
Nov 03 22:41:20.251127 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:20.251066    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51131709-120f-4b00-a6ec-34e879ac168b-istio-token\") pod \"e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq\" (UID: \"51131709-120f-4b00-a6ec-34e879ac168b\") " pod="openshift-ingress/e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq"
Nov 03 22:41:20.251293 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:20.251209    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmt69\" (UniqueName: \"kubernetes.io/projected/51131709-120f-4b00-a6ec-34e879ac168b-kube-api-access-pmt69\") pod \"e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq\" (UID: \"51131709-120f-4b00-a6ec-34e879ac168b\") " pod="openshift-ingress/e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq"
Nov 03 22:41:20.324916 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:20.322927    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:41:20.437952 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:20.437916    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq"
Nov 03 22:41:20.748142 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:20.748105    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:41:20.774870 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:20.774838    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq"]
Nov 03 22:41:20.790207 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:20.790162    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:41:20.790328 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:20.790283    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:41:20.790370 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:20.790329    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:41:21.105178 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:21.105094    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq" event={"ID":"51131709-120f-4b00-a6ec-34e879ac168b","Type":"ContainerStarted","Data":"33077418fd8a5ad61a910ab6c88db15081db2621f5b0591440c405b460d30f02"}
Nov 03 22:41:21.105178 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:21.105146    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq" event={"ID":"51131709-120f-4b00-a6ec-34e879ac168b","Type":"ContainerStarted","Data":"d550ba28fb462316335ae684aaf23bdb475a557d6b2a460633eecb70029369bb"}
Nov 03 22:41:21.169064 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:21.169000    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq" podStartSLOduration=2.168981122 podStartE2EDuration="2.168981122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:41:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:41:21.163713979 +0000 UTC m=+1705.189569842" watchObservedRunningTime="2026-11-03 22:41:21.168981122 +0000 UTC m=+1705.194836985"
Nov 03 22:41:21.438997 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:21.438956    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq"
Nov 03 22:41:21.446773 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:21.446749    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq"
Nov 03 22:41:21.858445 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:21.858347    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-71a7df-86bdf57769-s22zx"]
Nov 03 22:41:21.862818 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:21.862793    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-71a7df-86bdf57769-s22zx"
Nov 03 22:41:21.887911 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:21.887886    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-71a7df-86bdf57769-s22zx"]
Nov 03 22:41:21.890818 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:21.890799    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-71a7df-serving-cert\""
Nov 03 22:41:21.952253 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:21.952117    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddf32aeb-0884-4e84-aef8-19fa55f251e3-openshift-service-ca\") pod \"maas-api-e2e-inf-b-71a7df-86bdf57769-s22zx\" (UID: \"ddf32aeb-0884-4e84-aef8-19fa55f251e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-71a7df-86bdf57769-s22zx"
Nov 03 22:41:21.952253 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:21.952177    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddf32aeb-0884-4e84-aef8-19fa55f251e3-metrics-certs\") pod \"maas-api-e2e-inf-b-71a7df-86bdf57769-s22zx\" (UID: \"ddf32aeb-0884-4e84-aef8-19fa55f251e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-71a7df-86bdf57769-s22zx"
Nov 03 22:41:21.952528 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:21.952377    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpkfj\" (UniqueName: \"kubernetes.io/projected/ddf32aeb-0884-4e84-aef8-19fa55f251e3-kube-api-access-zpkfj\") pod \"maas-api-e2e-inf-b-71a7df-86bdf57769-s22zx\" (UID: \"ddf32aeb-0884-4e84-aef8-19fa55f251e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-71a7df-86bdf57769-s22zx"
Nov 03 22:41:21.956257 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:21.952651    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ddf32aeb-0884-4e84-aef8-19fa55f251e3-maas-api-tls\") pod \"maas-api-e2e-inf-b-71a7df-86bdf57769-s22zx\" (UID: \"ddf32aeb-0884-4e84-aef8-19fa55f251e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-71a7df-86bdf57769-s22zx"
Nov 03 22:41:21.984347 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:21.984311    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:41:22.058055 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:22.057958    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ddf32aeb-0884-4e84-aef8-19fa55f251e3-maas-api-tls\") pod \"maas-api-e2e-inf-b-71a7df-86bdf57769-s22zx\" (UID: \"ddf32aeb-0884-4e84-aef8-19fa55f251e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-71a7df-86bdf57769-s22zx"
Nov 03 22:41:22.058291 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:22.058082    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddf32aeb-0884-4e84-aef8-19fa55f251e3-openshift-service-ca\") pod \"maas-api-e2e-inf-b-71a7df-86bdf57769-s22zx\" (UID: \"ddf32aeb-0884-4e84-aef8-19fa55f251e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-71a7df-86bdf57769-s22zx"
Nov 03 22:41:22.058291 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:22.058121    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddf32aeb-0884-4e84-aef8-19fa55f251e3-metrics-certs\") pod \"maas-api-e2e-inf-b-71a7df-86bdf57769-s22zx\" (UID: \"ddf32aeb-0884-4e84-aef8-19fa55f251e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-71a7df-86bdf57769-s22zx"
Nov 03 22:41:22.058291 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:22.058237    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpkfj\" (UniqueName: \"kubernetes.io/projected/ddf32aeb-0884-4e84-aef8-19fa55f251e3-kube-api-access-zpkfj\") pod \"maas-api-e2e-inf-b-71a7df-86bdf57769-s22zx\" (UID: \"ddf32aeb-0884-4e84-aef8-19fa55f251e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-71a7df-86bdf57769-s22zx"
Nov 03 22:41:22.067380 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:22.067351    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddf32aeb-0884-4e84-aef8-19fa55f251e3-metrics-certs\") pod \"maas-api-e2e-inf-b-71a7df-86bdf57769-s22zx\" (UID: \"ddf32aeb-0884-4e84-aef8-19fa55f251e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-71a7df-86bdf57769-s22zx"
Nov 03 22:41:22.067723 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:22.067409    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddf32aeb-0884-4e84-aef8-19fa55f251e3-openshift-service-ca\") pod \"maas-api-e2e-inf-b-71a7df-86bdf57769-s22zx\" (UID: \"ddf32aeb-0884-4e84-aef8-19fa55f251e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-71a7df-86bdf57769-s22zx"
Nov 03 22:41:22.068890 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:22.068840    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ddf32aeb-0884-4e84-aef8-19fa55f251e3-maas-api-tls\") pod \"maas-api-e2e-inf-b-71a7df-86bdf57769-s22zx\" (UID: \"ddf32aeb-0884-4e84-aef8-19fa55f251e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-71a7df-86bdf57769-s22zx"
Nov 03 22:41:22.091097 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:22.091019    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpkfj\" (UniqueName: \"kubernetes.io/projected/ddf32aeb-0884-4e84-aef8-19fa55f251e3-kube-api-access-zpkfj\") pod \"maas-api-e2e-inf-b-71a7df-86bdf57769-s22zx\" (UID: \"ddf32aeb-0884-4e84-aef8-19fa55f251e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-71a7df-86bdf57769-s22zx"
Nov 03 22:41:22.115128 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:22.115047    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq"
Nov 03 22:41:22.116097 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:22.116073    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq"
Nov 03 22:41:22.131104 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:22.131068    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:41:22.177895 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:22.177860    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-71a7df-86bdf57769-s22zx"
Nov 03 22:41:22.345968 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:22.345936    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:41:22.522835 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:22.522806    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-71a7df-86bdf57769-s22zx"]
Nov 03 22:41:22.524047 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:41:22.524010    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddf32aeb_0884_4e84_aef8_19fa55f251e3.slice/crio-ffac2faf2a8e995d80af3cb5528267b72086dd677244f35559193fd3b49ac2ac WatchSource:0}: Error finding container ffac2faf2a8e995d80af3cb5528267b72086dd677244f35559193fd3b49ac2ac: Status 404 returned error can't find the container with id ffac2faf2a8e995d80af3cb5528267b72086dd677244f35559193fd3b49ac2ac
Nov 03 22:41:22.691894 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:22.691857    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:41:23.127838 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:23.127765    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-71a7df-86bdf57769-s22zx" event={"ID":"ddf32aeb-0884-4e84-aef8-19fa55f251e3","Type":"ContainerStarted","Data":"7fd45994dd6ff1dac740e065a8c8b408882cb175c7781a83411b4624f080a5ea"}
Nov 03 22:41:23.127838 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:23.127830    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-71a7df-86bdf57769-s22zx" event={"ID":"ddf32aeb-0884-4e84-aef8-19fa55f251e3","Type":"ContainerStarted","Data":"ffac2faf2a8e995d80af3cb5528267b72086dd677244f35559193fd3b49ac2ac"}
Nov 03 22:41:23.128778 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:23.128368    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-71a7df-86bdf57769-s22zx"
Nov 03 22:41:23.174257 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:23.174179    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-71a7df-86bdf57769-s22zx" podStartSLOduration=1.716185888 podStartE2EDuration="2.174159332s" totalImagesPullingTime="457.973444ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:41:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:41:23.169685805 +0000 UTC m=+1707.195541668" watchObservedRunningTime="2026-11-03 22:41:23.174159332 +0000 UTC m=+1707.200015195"
Nov 03 22:41:23.545119 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:23.545081    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:41:23.706032 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:23.705998    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:41:23.854846 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:23.854755    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:41:24.398239 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:24.391314    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:41:25.718016 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:25.717966    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:41:25.876050 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:25.876006    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:41:26.129209 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:26.129103    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:41:27.716541 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:27.716505    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:41:27.924035 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:27.923992    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:41:28.087252 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:28.087128    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:41:29.357850 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:29.357814    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:41:29.505572 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:29.505530    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-341c51-76d98b8f6c-cqm9p"]
Nov 03 22:41:29.510389 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:29.510365    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-341c51-76d98b8f6c-cqm9p"
Nov 03 22:41:29.525702 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:29.525675    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:41:29.530009 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:29.529981    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-341c51-76d98b8f6c-cqm9p"]
Nov 03 22:41:29.537553 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:29.537528    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-341c51\""
Nov 03 22:41:29.537685 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:29.537567    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-341c51-dockercfg-dvsdf\""
Nov 03 22:41:29.636288 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:29.635190    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16008341-844a-48a1-80dd-9f804fbcae4a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-341c51-76d98b8f6c-cqm9p\" (UID: \"16008341-844a-48a1-80dd-9f804fbcae4a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-341c51-76d98b8f6c-cqm9p"
Nov 03 22:41:29.636288 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:29.635531    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbt6m\" (UniqueName: \"kubernetes.io/projected/16008341-844a-48a1-80dd-9f804fbcae4a-kube-api-access-cbt6m\") pod \"payload-pre-processing-e2e-inf-a-341c51-76d98b8f6c-cqm9p\" (UID: \"16008341-844a-48a1-80dd-9f804fbcae4a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-341c51-76d98b8f6c-cqm9p"
Nov 03 22:41:29.660148 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:29.660108    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-341c51-857b566498-5d2tg"]
Nov 03 22:41:29.682852 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:29.682685    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-341c51-857b566498-5d2tg"
Nov 03 22:41:29.722847 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:29.722813    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-341c51-857b566498-5d2tg"]
Nov 03 22:41:29.752453 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:29.752408    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbt6m\" (UniqueName: \"kubernetes.io/projected/16008341-844a-48a1-80dd-9f804fbcae4a-kube-api-access-cbt6m\") pod \"payload-pre-processing-e2e-inf-a-341c51-76d98b8f6c-cqm9p\" (UID: \"16008341-844a-48a1-80dd-9f804fbcae4a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-341c51-76d98b8f6c-cqm9p"
Nov 03 22:41:29.753946 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:29.753437    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16008341-844a-48a1-80dd-9f804fbcae4a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-341c51-76d98b8f6c-cqm9p\" (UID: \"16008341-844a-48a1-80dd-9f804fbcae4a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-341c51-76d98b8f6c-cqm9p"
Nov 03 22:41:29.754460 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:29.754435    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16008341-844a-48a1-80dd-9f804fbcae4a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-341c51-76d98b8f6c-cqm9p\" (UID: \"16008341-844a-48a1-80dd-9f804fbcae4a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-341c51-76d98b8f6c-cqm9p"
Nov 03 22:41:29.775717 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:29.775688    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbt6m\" (UniqueName: \"kubernetes.io/projected/16008341-844a-48a1-80dd-9f804fbcae4a-kube-api-access-cbt6m\") pod \"payload-pre-processing-e2e-inf-a-341c51-76d98b8f6c-cqm9p\" (UID: \"16008341-844a-48a1-80dd-9f804fbcae4a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-341c51-76d98b8f6c-cqm9p"
Nov 03 22:41:29.791750 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:29.791717    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:41:29.825545 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:29.825507    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-341c51-76d98b8f6c-cqm9p"
Nov 03 22:41:29.854823 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:29.854783    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97ea71c9-692d-49fb-a079-183bda697421-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-341c51-857b566498-5d2tg\" (UID: \"97ea71c9-692d-49fb-a079-183bda697421\") " pod="openshift-ingress/payload-processing-e2e-inf-a-341c51-857b566498-5d2tg"
Nov 03 22:41:29.855088 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:29.855056    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkwgv\" (UniqueName: \"kubernetes.io/projected/97ea71c9-692d-49fb-a079-183bda697421-kube-api-access-xkwgv\") pod \"payload-processing-e2e-inf-a-341c51-857b566498-5d2tg\" (UID: \"97ea71c9-692d-49fb-a079-183bda697421\") " pod="openshift-ingress/payload-processing-e2e-inf-a-341c51-857b566498-5d2tg"
Nov 03 22:41:29.958439 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:29.957166    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkwgv\" (UniqueName: \"kubernetes.io/projected/97ea71c9-692d-49fb-a079-183bda697421-kube-api-access-xkwgv\") pod \"payload-processing-e2e-inf-a-341c51-857b566498-5d2tg\" (UID: \"97ea71c9-692d-49fb-a079-183bda697421\") " pod="openshift-ingress/payload-processing-e2e-inf-a-341c51-857b566498-5d2tg"
Nov 03 22:41:29.958439 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:29.957403    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97ea71c9-692d-49fb-a079-183bda697421-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-341c51-857b566498-5d2tg\" (UID: \"97ea71c9-692d-49fb-a079-183bda697421\") " pod="openshift-ingress/payload-processing-e2e-inf-a-341c51-857b566498-5d2tg"
Nov 03 22:41:29.959691 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:29.959669    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97ea71c9-692d-49fb-a079-183bda697421-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-341c51-857b566498-5d2tg\" (UID: \"97ea71c9-692d-49fb-a079-183bda697421\") " pod="openshift-ingress/payload-processing-e2e-inf-a-341c51-857b566498-5d2tg"
Nov 03 22:41:29.992179 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:29.992145    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkwgv\" (UniqueName: \"kubernetes.io/projected/97ea71c9-692d-49fb-a079-183bda697421-kube-api-access-xkwgv\") pod \"payload-processing-e2e-inf-a-341c51-857b566498-5d2tg\" (UID: \"97ea71c9-692d-49fb-a079-183bda697421\") " pod="openshift-ingress/payload-processing-e2e-inf-a-341c51-857b566498-5d2tg"
Nov 03 22:41:30.021147 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:30.021117    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-341c51-857b566498-5d2tg"
Nov 03 22:41:30.148530 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:30.148498    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-71a7df-86bdf57769-s22zx"
Nov 03 22:41:30.154118 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:30.154090    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-341c51-76d98b8f6c-cqm9p"]
Nov 03 22:41:30.155791 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:41:30.155754    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16008341_844a_48a1_80dd_9f804fbcae4a.slice/crio-f37d2f5c7798eb28ff2ab2746a943e66a2784f8a8332d381791d092718e88401 WatchSource:0}: Error finding container f37d2f5c7798eb28ff2ab2746a943e66a2784f8a8332d381791d092718e88401: Status 404 returned error can't find the container with id f37d2f5c7798eb28ff2ab2746a943e66a2784f8a8332d381791d092718e88401
Nov 03 22:41:30.182631 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:30.182603    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-341c51-76d98b8f6c-cqm9p" event={"ID":"16008341-844a-48a1-80dd-9f804fbcae4a","Type":"ContainerStarted","Data":"f37d2f5c7798eb28ff2ab2746a943e66a2784f8a8332d381791d092718e88401"}
Nov 03 22:41:30.339481 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:30.339446    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-341c51-857b566498-5d2tg"]
Nov 03 22:41:30.340357 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:41:30.340326    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97ea71c9_692d_49fb_a079_183bda697421.slice/crio-42b77a26829cd07e9133b01dd6258056cea9d7328cb4984695a1db45885eef8d WatchSource:0}: Error finding container 42b77a26829cd07e9133b01dd6258056cea9d7328cb4984695a1db45885eef8d: Status 404 returned error can't find the container with id 42b77a26829cd07e9133b01dd6258056cea9d7328cb4984695a1db45885eef8d
Nov 03 22:41:31.191171 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:31.191136    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-341c51-76d98b8f6c-cqm9p" event={"ID":"16008341-844a-48a1-80dd-9f804fbcae4a","Type":"ContainerStarted","Data":"409774e8ed984be083cbd48927211d23ba434226afca97ce4858a73fa9c8003e"}
Nov 03 22:41:31.191732 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:31.191242    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-341c51-76d98b8f6c-cqm9p"
Nov 03 22:41:31.192883 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:31.192856    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-341c51-857b566498-5d2tg" event={"ID":"97ea71c9-692d-49fb-a079-183bda697421","Type":"ContainerStarted","Data":"f285c04e5ad7f1b0e10d0bf65034d1717b76113e91a72f5b07a7fa8804242eec"}
Nov 03 22:41:31.192997 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:31.192900    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-341c51-857b566498-5d2tg" event={"ID":"97ea71c9-692d-49fb-a079-183bda697421","Type":"ContainerStarted","Data":"42b77a26829cd07e9133b01dd6258056cea9d7328cb4984695a1db45885eef8d"}
Nov 03 22:41:31.192997 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:31.192982    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-341c51-857b566498-5d2tg"
Nov 03 22:41:31.233240 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:31.233166    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-341c51-76d98b8f6c-cqm9p" podStartSLOduration=2.2331521 podStartE2EDuration="2.2331521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:41:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:41:31.229839718 +0000 UTC m=+1715.255695579" watchObservedRunningTime="2026-11-03 22:41:31.2331521 +0000 UTC m=+1715.259007962"
Nov 03 22:41:31.260184 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:41:31.260152    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:41:31.269738 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:31.269693    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-341c51-857b566498-5d2tg" podStartSLOduration=2.269678467 podStartE2EDuration="2.269678467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:41:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:41:31.268722468 +0000 UTC m=+1715.294578329" watchObservedRunningTime="2026-11-03 22:41:31.269678467 +0000 UTC m=+1715.295534329"
Nov 03 22:41:31.513431 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:31.513305    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:41:31.705755 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:31.705724    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:41:31.846305 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:31.846232    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:41:33.431279 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:33.431247    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:41:35.101924 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:35.101895    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:41:36.600883 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:36.600845    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:41:38.130802 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:38.130763    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:41:39.850283 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:39.837622    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:41:41.390035 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:41.390003    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:41:42.206366 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:42.206338    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-341c51-857b566498-5d2tg"
Nov 03 22:41:42.206656 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:42.206632    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-341c51-76d98b8f6c-cqm9p"
Nov 03 22:41:42.861957 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:42.861919    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:41:44.456203 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:44.456156    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:41:46.085384 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:46.085351    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:41:47.599097 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:47.599042    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:41:47.743918 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:47.743872    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:41:49.217697 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:49.217667    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:41:50.706069 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:50.706032    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:41:52.214116 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:52.214083    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:41:52.702134 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:52.702101    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:41:54.006288 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:54.006192    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:41:54.696416 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:54.696375    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:41:55.133547 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:55.133461    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:41:55.539036 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:55.539002    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz"]
Nov 03 22:41:55.547333 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:55.547305    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz"
Nov 03 22:41:55.578409 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:55.578379    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-ccdb60-openshift-default-dockercfg-q8nbf\""
Nov 03 22:41:55.612953 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:55.612921    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz"]
Nov 03 22:41:55.642950 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:55.642879    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:41:55.754056 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:55.753938    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79d7b9fc-6282-4aa1-bae1-e8cbe554eadf-credential-socket\") pod \"e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz\" (UID: \"79d7b9fc-6282-4aa1-bae1-e8cbe554eadf\") " pod="openshift-ingress/e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz"
Nov 03 22:41:55.754512 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:55.754487    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79d7b9fc-6282-4aa1-bae1-e8cbe554eadf-istio-envoy\") pod \"e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz\" (UID: \"79d7b9fc-6282-4aa1-bae1-e8cbe554eadf\") " pod="openshift-ingress/e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz"
Nov 03 22:41:55.754736 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:55.754717    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxdqt\" (UniqueName: \"kubernetes.io/projected/79d7b9fc-6282-4aa1-bae1-e8cbe554eadf-kube-api-access-hxdqt\") pod \"e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz\" (UID: \"79d7b9fc-6282-4aa1-bae1-e8cbe554eadf\") " pod="openshift-ingress/e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz"
Nov 03 22:41:55.754937 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:55.754921    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79d7b9fc-6282-4aa1-bae1-e8cbe554eadf-workload-certs\") pod \"e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz\" (UID: \"79d7b9fc-6282-4aa1-bae1-e8cbe554eadf\") " pod="openshift-ingress/e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz"
Nov 03 22:41:55.755155 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:55.755138    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79d7b9fc-6282-4aa1-bae1-e8cbe554eadf-istio-token\") pod \"e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz\" (UID: \"79d7b9fc-6282-4aa1-bae1-e8cbe554eadf\") " pod="openshift-ingress/e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz"
Nov 03 22:41:55.755459 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:55.755440    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79d7b9fc-6282-4aa1-bae1-e8cbe554eadf-workload-socket\") pod \"e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz\" (UID: \"79d7b9fc-6282-4aa1-bae1-e8cbe554eadf\") " pod="openshift-ingress/e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz"
Nov 03 22:41:55.755662 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:55.755640    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79d7b9fc-6282-4aa1-bae1-e8cbe554eadf-istio-data\") pod \"e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz\" (UID: \"79d7b9fc-6282-4aa1-bae1-e8cbe554eadf\") " pod="openshift-ingress/e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz"
Nov 03 22:41:55.755841 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:55.755825    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79d7b9fc-6282-4aa1-bae1-e8cbe554eadf-istio-podinfo\") pod \"e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz\" (UID: \"79d7b9fc-6282-4aa1-bae1-e8cbe554eadf\") " pod="openshift-ingress/e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz"
Nov 03 22:41:55.756111 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:55.756091    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79d7b9fc-6282-4aa1-bae1-e8cbe554eadf-istiod-ca-cert\") pod \"e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz\" (UID: \"79d7b9fc-6282-4aa1-bae1-e8cbe554eadf\") " pod="openshift-ingress/e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz"
Nov 03 22:41:55.882360 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:55.882269    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79d7b9fc-6282-4aa1-bae1-e8cbe554eadf-workload-socket\") pod \"e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz\" (UID: \"79d7b9fc-6282-4aa1-bae1-e8cbe554eadf\") " pod="openshift-ingress/e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz"
Nov 03 22:41:55.890544 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:55.890318    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79d7b9fc-6282-4aa1-bae1-e8cbe554eadf-istio-data\") pod \"e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz\" (UID: \"79d7b9fc-6282-4aa1-bae1-e8cbe554eadf\") " pod="openshift-ingress/e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz"
Nov 03 22:41:55.890820 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:55.890795    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79d7b9fc-6282-4aa1-bae1-e8cbe554eadf-istio-podinfo\") pod \"e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz\" (UID: \"79d7b9fc-6282-4aa1-bae1-e8cbe554eadf\") " pod="openshift-ingress/e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz"
Nov 03 22:41:55.890995 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:55.890977    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79d7b9fc-6282-4aa1-bae1-e8cbe554eadf-istiod-ca-cert\") pod \"e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz\" (UID: \"79d7b9fc-6282-4aa1-bae1-e8cbe554eadf\") " pod="openshift-ingress/e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz"
Nov 03 22:41:55.900247 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:55.891288    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79d7b9fc-6282-4aa1-bae1-e8cbe554eadf-credential-socket\") pod \"e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz\" (UID: \"79d7b9fc-6282-4aa1-bae1-e8cbe554eadf\") " pod="openshift-ingress/e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz"
Nov 03 22:41:55.900588 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:55.900557    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79d7b9fc-6282-4aa1-bae1-e8cbe554eadf-istio-envoy\") pod \"e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz\" (UID: \"79d7b9fc-6282-4aa1-bae1-e8cbe554eadf\") " pod="openshift-ingress/e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz"
Nov 03 22:41:55.900740 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:55.900720    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxdqt\" (UniqueName: \"kubernetes.io/projected/79d7b9fc-6282-4aa1-bae1-e8cbe554eadf-kube-api-access-hxdqt\") pod \"e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz\" (UID: \"79d7b9fc-6282-4aa1-bae1-e8cbe554eadf\") " pod="openshift-ingress/e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz"
Nov 03 22:41:55.900891 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:55.900875    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79d7b9fc-6282-4aa1-bae1-e8cbe554eadf-workload-certs\") pod \"e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz\" (UID: \"79d7b9fc-6282-4aa1-bae1-e8cbe554eadf\") " pod="openshift-ingress/e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz"
Nov 03 22:41:55.901016 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:55.900998    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79d7b9fc-6282-4aa1-bae1-e8cbe554eadf-istio-token\") pod \"e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz\" (UID: \"79d7b9fc-6282-4aa1-bae1-e8cbe554eadf\") " pod="openshift-ingress/e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz"
Nov 03 22:41:55.911241 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:55.910985    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79d7b9fc-6282-4aa1-bae1-e8cbe554eadf-istiod-ca-cert\") pod \"e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz\" (UID: \"79d7b9fc-6282-4aa1-bae1-e8cbe554eadf\") " pod="openshift-ingress/e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz"
Nov 03 22:41:55.911241 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:55.889685    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79d7b9fc-6282-4aa1-bae1-e8cbe554eadf-workload-socket\") pod \"e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz\" (UID: \"79d7b9fc-6282-4aa1-bae1-e8cbe554eadf\") " pod="openshift-ingress/e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz"
Nov 03 22:41:55.911480 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:55.911310    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79d7b9fc-6282-4aa1-bae1-e8cbe554eadf-istio-data\") pod \"e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz\" (UID: \"79d7b9fc-6282-4aa1-bae1-e8cbe554eadf\") " pod="openshift-ingress/e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz"
Nov 03 22:41:55.912083 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:55.912054    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79d7b9fc-6282-4aa1-bae1-e8cbe554eadf-credential-socket\") pod \"e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz\" (UID: \"79d7b9fc-6282-4aa1-bae1-e8cbe554eadf\") " pod="openshift-ingress/e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz"
Nov 03 22:41:55.945331 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:55.921654    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79d7b9fc-6282-4aa1-bae1-e8cbe554eadf-workload-certs\") pod \"e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz\" (UID: \"79d7b9fc-6282-4aa1-bae1-e8cbe554eadf\") " pod="openshift-ingress/e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz"
Nov 03 22:41:55.964497 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:55.948910    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79d7b9fc-6282-4aa1-bae1-e8cbe554eadf-istio-podinfo\") pod \"e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz\" (UID: \"79d7b9fc-6282-4aa1-bae1-e8cbe554eadf\") " pod="openshift-ingress/e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz"
Nov 03 22:41:55.977681 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:55.976262    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79d7b9fc-6282-4aa1-bae1-e8cbe554eadf-istio-envoy\") pod \"e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz\" (UID: \"79d7b9fc-6282-4aa1-bae1-e8cbe554eadf\") " pod="openshift-ingress/e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz"
Nov 03 22:41:55.999863 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:55.999791    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79d7b9fc-6282-4aa1-bae1-e8cbe554eadf-istio-token\") pod \"e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz\" (UID: \"79d7b9fc-6282-4aa1-bae1-e8cbe554eadf\") " pod="openshift-ingress/e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz"
Nov 03 22:41:56.029059 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:56.029027    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxdqt\" (UniqueName: \"kubernetes.io/projected/79d7b9fc-6282-4aa1-bae1-e8cbe554eadf-kube-api-access-hxdqt\") pod \"e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz\" (UID: \"79d7b9fc-6282-4aa1-bae1-e8cbe554eadf\") " pod="openshift-ingress/e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz"
Nov 03 22:41:56.324374 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:56.310562    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-ccdb60-openshift-default-dockercfg-q8nbf\""
Nov 03 22:41:56.324970 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:56.324584    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz"
Nov 03 22:41:56.643241 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:56.630701    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:41:56.838091 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:41:56.838056    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79d7b9fc_6282_4aa1_bae1_e8cbe554eadf.slice/crio-b3cee509e6e8fcd7ec4963516e6ecdecb824606ea1055bb05cb40e6a5d2ddbae WatchSource:0}: Error finding container b3cee509e6e8fcd7ec4963516e6ecdecb824606ea1055bb05cb40e6a5d2ddbae: Status 404 returned error can't find the container with id b3cee509e6e8fcd7ec4963516e6ecdecb824606ea1055bb05cb40e6a5d2ddbae
Nov 03 22:41:56.838211 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:56.838177    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz"]
Nov 03 22:41:56.842195 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:56.842147    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:41:56.842322 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:56.842258    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:41:56.842322 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:56.842303    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:41:56.961316 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:56.961281    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:41:57.490027 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:57.489984    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz" event={"ID":"79d7b9fc-6282-4aa1-bae1-e8cbe554eadf","Type":"ContainerStarted","Data":"cf8ef89b306aecedf3c8c429d71f4a272f85645abec9c84d410be06c528f3acd"}
Nov 03 22:41:57.490564 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:57.490043    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz" event={"ID":"79d7b9fc-6282-4aa1-bae1-e8cbe554eadf","Type":"ContainerStarted","Data":"b3cee509e6e8fcd7ec4963516e6ecdecb824606ea1055bb05cb40e6a5d2ddbae"}
Nov 03 22:41:57.547365 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:57.547298    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz" podStartSLOduration=2.547279901 podStartE2EDuration="2.547279901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:41:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:41:57.545497762 +0000 UTC m=+1741.571353626" watchObservedRunningTime="2026-11-03 22:41:57.547279901 +0000 UTC m=+1741.573135763"
Nov 03 22:41:58.325034 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:58.324995    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz"
Nov 03 22:41:58.331165 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:58.331138    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz"
Nov 03 22:41:58.498263 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:58.498230    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz"
Nov 03 22:41:58.499274 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:58.499255    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz"
Nov 03 22:41:58.613947 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:41:58.613863    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:42:00.127231 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:00.127191    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:42:00.293983 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:00.293947    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:42:00.437774 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:00.437739    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:42:01.261099 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:42:01.261064    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:42:02.129723 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:02.129690    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:42:03.824740 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:03.824703    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:42:04.217341 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:04.217302    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:42:04.345293 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:04.345258    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ccdb60-779c4b9f95-rrp8q"]
Nov 03 22:42:04.351620 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:04.351597    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ccdb60-779c4b9f95-rrp8q"
Nov 03 22:42:04.366808 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:04.366782    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ccdb60-779c4b9f95-rrp8q"]
Nov 03 22:42:04.369523 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:04.369486    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-ccdb60\""
Nov 03 22:42:04.371950 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:04.371929    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:42:04.373783 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:04.373765    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-ccdb60-dockercfg-64t45\""
Nov 03 22:42:04.517810 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:04.502622    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xqcq\" (UniqueName: \"kubernetes.io/projected/52161b5e-fa32-4d5c-9695-b8fd60d6b87a-kube-api-access-2xqcq\") pod \"payload-pre-processing-e2e-inf-b-ccdb60-779c4b9f95-rrp8q\" (UID: \"52161b5e-fa32-4d5c-9695-b8fd60d6b87a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ccdb60-779c4b9f95-rrp8q"
Nov 03 22:42:04.518490 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:04.518451    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52161b5e-fa32-4d5c-9695-b8fd60d6b87a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ccdb60-779c4b9f95-rrp8q\" (UID: \"52161b5e-fa32-4d5c-9695-b8fd60d6b87a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ccdb60-779c4b9f95-rrp8q"
Nov 03 22:42:04.622568 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:04.622525    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52161b5e-fa32-4d5c-9695-b8fd60d6b87a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ccdb60-779c4b9f95-rrp8q\" (UID: \"52161b5e-fa32-4d5c-9695-b8fd60d6b87a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ccdb60-779c4b9f95-rrp8q"
Nov 03 22:42:04.622757 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:04.622639    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xqcq\" (UniqueName: \"kubernetes.io/projected/52161b5e-fa32-4d5c-9695-b8fd60d6b87a-kube-api-access-2xqcq\") pod \"payload-pre-processing-e2e-inf-b-ccdb60-779c4b9f95-rrp8q\" (UID: \"52161b5e-fa32-4d5c-9695-b8fd60d6b87a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ccdb60-779c4b9f95-rrp8q"
Nov 03 22:42:04.624045 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:04.624006    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52161b5e-fa32-4d5c-9695-b8fd60d6b87a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ccdb60-779c4b9f95-rrp8q\" (UID: \"52161b5e-fa32-4d5c-9695-b8fd60d6b87a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ccdb60-779c4b9f95-rrp8q"
Nov 03 22:42:04.664032 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:04.663987    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xqcq\" (UniqueName: \"kubernetes.io/projected/52161b5e-fa32-4d5c-9695-b8fd60d6b87a-kube-api-access-2xqcq\") pod \"payload-pre-processing-e2e-inf-b-ccdb60-779c4b9f95-rrp8q\" (UID: \"52161b5e-fa32-4d5c-9695-b8fd60d6b87a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ccdb60-779c4b9f95-rrp8q"
Nov 03 22:42:04.669512 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:04.669479    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ccdb60-779c4b9f95-rrp8q"
Nov 03 22:42:05.007245 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:05.007186    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ccdb60-779c4b9f95-rrp8q"]
Nov 03 22:42:05.008206 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:42:05.008181    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52161b5e_fa32_4d5c_9695_b8fd60d6b87a.slice/crio-eadf4fed6949863c016c0b55bcfc85f0965328f6a91632de4a6d30cf6b28c7a0 WatchSource:0}: Error finding container eadf4fed6949863c016c0b55bcfc85f0965328f6a91632de4a6d30cf6b28c7a0: Status 404 returned error can't find the container with id eadf4fed6949863c016c0b55bcfc85f0965328f6a91632de4a6d30cf6b28c7a0
Nov 03 22:42:05.582235 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:05.581796    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ccdb60-779c4b9f95-rrp8q" event={"ID":"52161b5e-fa32-4d5c-9695-b8fd60d6b87a","Type":"ContainerStarted","Data":"e420118dd5786a44b15110d21b74b96dc29dd1fd06e0f0f0d99af07b976b64eb"}
Nov 03 22:42:05.582235 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:05.581863    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ccdb60-779c4b9f95-rrp8q" event={"ID":"52161b5e-fa32-4d5c-9695-b8fd60d6b87a","Type":"ContainerStarted","Data":"eadf4fed6949863c016c0b55bcfc85f0965328f6a91632de4a6d30cf6b28c7a0"}
Nov 03 22:42:05.582235 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:05.581894    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ccdb60-779c4b9f95-rrp8q"
Nov 03 22:42:05.629022 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:05.628909    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ccdb60-779c4b9f95-rrp8q" podStartSLOduration=1.6288941879999999 podStartE2EDuration="1.628894188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:42:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:42:05.626307396 +0000 UTC m=+1749.652163261" watchObservedRunningTime="2026-11-03 22:42:05.628894188 +0000 UTC m=+1749.654750049"
Nov 03 22:42:06.014132 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:06.014095    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:42:06.157616 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:06.157496    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:42:07.773308 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:07.773279    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:42:08.979909 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:08.979875    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv"]
Nov 03 22:42:08.984452 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:08.984427    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv"
Nov 03 22:42:09.004853 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:09.004820    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8b1f74b-851b-41ed-a643-f01042642e59-credential-socket\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv\" (UID: \"f8b1f74b-851b-41ed-a643-f01042642e59\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv"
Nov 03 22:42:09.005022 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:09.004888    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7plt\" (UniqueName: \"kubernetes.io/projected/f8b1f74b-851b-41ed-a643-f01042642e59-kube-api-access-w7plt\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv\" (UID: \"f8b1f74b-851b-41ed-a643-f01042642e59\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv"
Nov 03 22:42:09.005117 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:09.005095    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8b1f74b-851b-41ed-a643-f01042642e59-workload-socket\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv\" (UID: \"f8b1f74b-851b-41ed-a643-f01042642e59\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv"
Nov 03 22:42:09.005178 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:09.005144    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8b1f74b-851b-41ed-a643-f01042642e59-istio-data\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv\" (UID: \"f8b1f74b-851b-41ed-a643-f01042642e59\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv"
Nov 03 22:42:09.005350 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:09.005331    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8b1f74b-851b-41ed-a643-f01042642e59-istio-envoy\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv\" (UID: \"f8b1f74b-851b-41ed-a643-f01042642e59\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv"
Nov 03 22:42:09.005431 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:09.005373    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8b1f74b-851b-41ed-a643-f01042642e59-workload-certs\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv\" (UID: \"f8b1f74b-851b-41ed-a643-f01042642e59\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv"
Nov 03 22:42:09.005431 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:09.005390    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8b1f74b-851b-41ed-a643-f01042642e59-istio-podinfo\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv\" (UID: \"f8b1f74b-851b-41ed-a643-f01042642e59\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv"
Nov 03 22:42:09.005550 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:09.005507    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8b1f74b-851b-41ed-a643-f01042642e59-istiod-ca-cert\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv\" (UID: \"f8b1f74b-851b-41ed-a643-f01042642e59\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv"
Nov 03 22:42:09.005613 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:09.005549    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8b1f74b-851b-41ed-a643-f01042642e59-istio-token\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv\" (UID: \"f8b1f74b-851b-41ed-a643-f01042642e59\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv"
Nov 03 22:42:09.005894 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:09.005874    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv"]
Nov 03 22:42:09.013514 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:09.013480    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-71a7df-openshift-default-dockercfg-q4mt7\""
Nov 03 22:42:09.106752 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:09.106720    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8b1f74b-851b-41ed-a643-f01042642e59-workload-socket\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv\" (UID: \"f8b1f74b-851b-41ed-a643-f01042642e59\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv"
Nov 03 22:42:09.106959 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:09.106765    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8b1f74b-851b-41ed-a643-f01042642e59-istio-data\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv\" (UID: \"f8b1f74b-851b-41ed-a643-f01042642e59\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv"
Nov 03 22:42:09.106959 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:09.106878    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8b1f74b-851b-41ed-a643-f01042642e59-istio-envoy\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv\" (UID: \"f8b1f74b-851b-41ed-a643-f01042642e59\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv"
Nov 03 22:42:09.106959 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:09.106935    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8b1f74b-851b-41ed-a643-f01042642e59-workload-certs\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv\" (UID: \"f8b1f74b-851b-41ed-a643-f01042642e59\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv"
Nov 03 22:42:09.107126 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:09.106961    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8b1f74b-851b-41ed-a643-f01042642e59-istio-podinfo\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv\" (UID: \"f8b1f74b-851b-41ed-a643-f01042642e59\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv"
Nov 03 22:42:09.107126 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:09.107105    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8b1f74b-851b-41ed-a643-f01042642e59-istiod-ca-cert\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv\" (UID: \"f8b1f74b-851b-41ed-a643-f01042642e59\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv"
Nov 03 22:42:09.107273 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:09.107144    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8b1f74b-851b-41ed-a643-f01042642e59-workload-socket\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv\" (UID: \"f8b1f74b-851b-41ed-a643-f01042642e59\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv"
Nov 03 22:42:09.107273 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:09.107157    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8b1f74b-851b-41ed-a643-f01042642e59-istio-token\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv\" (UID: \"f8b1f74b-851b-41ed-a643-f01042642e59\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv"
Nov 03 22:42:09.107273 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:09.107204    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8b1f74b-851b-41ed-a643-f01042642e59-istio-data\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv\" (UID: \"f8b1f74b-851b-41ed-a643-f01042642e59\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv"
Nov 03 22:42:09.107452 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:09.107316    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8b1f74b-851b-41ed-a643-f01042642e59-credential-socket\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv\" (UID: \"f8b1f74b-851b-41ed-a643-f01042642e59\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv"
Nov 03 22:42:09.107452 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:09.107353    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8b1f74b-851b-41ed-a643-f01042642e59-workload-certs\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv\" (UID: \"f8b1f74b-851b-41ed-a643-f01042642e59\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv"
Nov 03 22:42:09.107452 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:09.107373    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7plt\" (UniqueName: \"kubernetes.io/projected/f8b1f74b-851b-41ed-a643-f01042642e59-kube-api-access-w7plt\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv\" (UID: \"f8b1f74b-851b-41ed-a643-f01042642e59\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv"
Nov 03 22:42:09.107639 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:09.107617    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8b1f74b-851b-41ed-a643-f01042642e59-credential-socket\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv\" (UID: \"f8b1f74b-851b-41ed-a643-f01042642e59\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv"
Nov 03 22:42:09.108038 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:09.108010    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8b1f74b-851b-41ed-a643-f01042642e59-istiod-ca-cert\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv\" (UID: \"f8b1f74b-851b-41ed-a643-f01042642e59\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv"
Nov 03 22:42:09.109389 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:09.109360    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8b1f74b-851b-41ed-a643-f01042642e59-istio-envoy\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv\" (UID: \"f8b1f74b-851b-41ed-a643-f01042642e59\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv"
Nov 03 22:42:09.109999 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:09.109981    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8b1f74b-851b-41ed-a643-f01042642e59-istio-podinfo\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv\" (UID: \"f8b1f74b-851b-41ed-a643-f01042642e59\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv"
Nov 03 22:42:09.131526 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:09.131500    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8b1f74b-851b-41ed-a643-f01042642e59-istio-token\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv\" (UID: \"f8b1f74b-851b-41ed-a643-f01042642e59\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv"
Nov 03 22:42:09.131642 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:09.131591    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7plt\" (UniqueName: \"kubernetes.io/projected/f8b1f74b-851b-41ed-a643-f01042642e59-kube-api-access-w7plt\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv\" (UID: \"f8b1f74b-851b-41ed-a643-f01042642e59\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv"
Nov 03 22:42:09.293776 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:09.293693    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:42:09.299341 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:09.299320    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv"
Nov 03 22:42:09.442548 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:09.442504    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:42:09.636328 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:09.636298    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv"]
Nov 03 22:42:09.636714 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:42:09.636692    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8b1f74b_851b_41ed_a643_f01042642e59.slice/crio-c3d5366be22c353713cb7dc95c4988637d5b891d68fa5d025569ad3ed4d56275 WatchSource:0}: Error finding container c3d5366be22c353713cb7dc95c4988637d5b891d68fa5d025569ad3ed4d56275: Status 404 returned error can't find the container with id c3d5366be22c353713cb7dc95c4988637d5b891d68fa5d025569ad3ed4d56275
Nov 03 22:42:09.639597 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:09.639563    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:42:09.639711 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:09.639643    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:42:09.639711 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:09.639685    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:42:10.830263 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:10.829445    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv" event={"ID":"f8b1f74b-851b-41ed-a643-f01042642e59","Type":"ContainerStarted","Data":"654b952ec5cff6ad73e51965d40370e209d7e9eed9c39f510d8348cbe95ab7b2"}
Nov 03 22:42:10.830933 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:10.830900    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv" event={"ID":"f8b1f74b-851b-41ed-a643-f01042642e59","Type":"ContainerStarted","Data":"c3d5366be22c353713cb7dc95c4988637d5b891d68fa5d025569ad3ed4d56275"}
Nov 03 22:42:10.909041 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:10.908983    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv" podStartSLOduration=2.908965534 podStartE2EDuration="2.908965534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:42:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:42:10.894575259 +0000 UTC m=+1754.920431122" watchObservedRunningTime="2026-11-03 22:42:10.908965534 +0000 UTC m=+1754.934821396"
Nov 03 22:42:11.299903 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:11.299870    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv"
Nov 03 22:42:11.301023 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:11.300991    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:42:11.301482 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:11.301461    4431 patch_prober.go:29] interesting pod/e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Nov 03 22:42:11.301548 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:11.301520    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv" podUID="f8b1f74b-851b-41ed-a643-f01042642e59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Nov 03 22:42:11.437521 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:11.437488    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:42:11.714854 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:11.714818    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:42:12.306243 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:12.301437    4431 patch_prober.go:29] interesting pod/e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Nov 03 22:42:12.306243 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:12.301521    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv" podUID="f8b1f74b-851b-41ed-a643-f01042642e59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Nov 03 22:42:13.125019 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:13.122800    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:42:13.300061 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:13.300015    4431 patch_prober.go:29] interesting pod/e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Nov 03 22:42:13.300271 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:13.300095    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv" podUID="f8b1f74b-851b-41ed-a643-f01042642e59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Nov 03 22:42:14.299818 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:14.299781    4431 patch_prober.go:29] interesting pod/e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Nov 03 22:42:14.300365 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:14.299854    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv" podUID="f8b1f74b-851b-41ed-a643-f01042642e59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Nov 03 22:42:15.015276 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:15.015240    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:42:15.300583 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:15.300500    4431 patch_prober.go:29] interesting pod/e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Nov 03 22:42:15.300583 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:15.300559    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv" podUID="f8b1f74b-851b-41ed-a643-f01042642e59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Nov 03 22:42:15.488528 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:15.488490    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:42:16.308279 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:16.308246    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv"
Nov 03 22:42:16.308764 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:16.308618    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv"
Nov 03 22:42:16.309338 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:16.309317    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv"
Nov 03 22:42:16.592235 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:16.592195    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ccdb60-779c4b9f95-rrp8q"
Nov 03 22:42:16.894838 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:16.894752    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:42:17.842717 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:17.842684    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-71a7df/test-model-71a7df-kserve-6967965559-nd2vz"]
Nov 03 22:42:17.847622 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:17.847596    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-71a7df/test-model-71a7df-kserve-6967965559-nd2vz"
Nov 03 22:42:17.866171 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:17.866139    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-71a7df/test-model-71a7df-kserve-6967965559-nd2vz"]
Nov 03 22:42:17.873356 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:17.873326    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-71a7df\"/\"default-dockercfg-dr8c9\""
Nov 03 22:42:17.873477 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:17.873363    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-71a7df\"/\"test-model-71a7df-kserve-self-signed-certs\""
Nov 03 22:42:17.878531 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:17.878509    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-71a7df\"/\"openshift-service-ca.crt\""
Nov 03 22:42:17.878666 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:17.878547    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-71a7df\"/\"kube-root-ca.crt\""
Nov 03 22:42:17.916353 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:17.916047    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1018108a-ceed-4fd7-8ad8-23f7478d7cae-model-cache\") pod \"test-model-71a7df-kserve-6967965559-nd2vz\" (UID: \"1018108a-ceed-4fd7-8ad8-23f7478d7cae\") " pod="ai-tenant-e2e-inf-b-71a7df/test-model-71a7df-kserve-6967965559-nd2vz"
Nov 03 22:42:17.916353 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:17.916280    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1018108a-ceed-4fd7-8ad8-23f7478d7cae-home\") pod \"test-model-71a7df-kserve-6967965559-nd2vz\" (UID: \"1018108a-ceed-4fd7-8ad8-23f7478d7cae\") " pod="ai-tenant-e2e-inf-b-71a7df/test-model-71a7df-kserve-6967965559-nd2vz"
Nov 03 22:42:17.916603 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:17.916423    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1018108a-ceed-4fd7-8ad8-23f7478d7cae-tls-certs\") pod \"test-model-71a7df-kserve-6967965559-nd2vz\" (UID: \"1018108a-ceed-4fd7-8ad8-23f7478d7cae\") " pod="ai-tenant-e2e-inf-b-71a7df/test-model-71a7df-kserve-6967965559-nd2vz"
Nov 03 22:42:17.916673 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:17.916624    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1018108a-ceed-4fd7-8ad8-23f7478d7cae-tmp-dir\") pod \"test-model-71a7df-kserve-6967965559-nd2vz\" (UID: \"1018108a-ceed-4fd7-8ad8-23f7478d7cae\") " pod="ai-tenant-e2e-inf-b-71a7df/test-model-71a7df-kserve-6967965559-nd2vz"
Nov 03 22:42:17.916755 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:17.916721    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1018108a-ceed-4fd7-8ad8-23f7478d7cae-dshm\") pod \"test-model-71a7df-kserve-6967965559-nd2vz\" (UID: \"1018108a-ceed-4fd7-8ad8-23f7478d7cae\") " pod="ai-tenant-e2e-inf-b-71a7df/test-model-71a7df-kserve-6967965559-nd2vz"
Nov 03 22:42:17.919594 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:17.919566    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swhwv\" (UniqueName: \"kubernetes.io/projected/1018108a-ceed-4fd7-8ad8-23f7478d7cae-kube-api-access-swhwv\") pod \"test-model-71a7df-kserve-6967965559-nd2vz\" (UID: \"1018108a-ceed-4fd7-8ad8-23f7478d7cae\") " pod="ai-tenant-e2e-inf-b-71a7df/test-model-71a7df-kserve-6967965559-nd2vz"
Nov 03 22:42:17.934468 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:17.934416    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:42:18.022033 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:18.021785    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1018108a-ceed-4fd7-8ad8-23f7478d7cae-tmp-dir\") pod \"test-model-71a7df-kserve-6967965559-nd2vz\" (UID: \"1018108a-ceed-4fd7-8ad8-23f7478d7cae\") " pod="ai-tenant-e2e-inf-b-71a7df/test-model-71a7df-kserve-6967965559-nd2vz"
Nov 03 22:42:18.022033 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:18.021926    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1018108a-ceed-4fd7-8ad8-23f7478d7cae-dshm\") pod \"test-model-71a7df-kserve-6967965559-nd2vz\" (UID: \"1018108a-ceed-4fd7-8ad8-23f7478d7cae\") " pod="ai-tenant-e2e-inf-b-71a7df/test-model-71a7df-kserve-6967965559-nd2vz"
Nov 03 22:42:18.022343 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:18.022164    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swhwv\" (UniqueName: \"kubernetes.io/projected/1018108a-ceed-4fd7-8ad8-23f7478d7cae-kube-api-access-swhwv\") pod \"test-model-71a7df-kserve-6967965559-nd2vz\" (UID: \"1018108a-ceed-4fd7-8ad8-23f7478d7cae\") " pod="ai-tenant-e2e-inf-b-71a7df/test-model-71a7df-kserve-6967965559-nd2vz"
Nov 03 22:42:18.022343 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:18.022189    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1018108a-ceed-4fd7-8ad8-23f7478d7cae-tmp-dir\") pod \"test-model-71a7df-kserve-6967965559-nd2vz\" (UID: \"1018108a-ceed-4fd7-8ad8-23f7478d7cae\") " pod="ai-tenant-e2e-inf-b-71a7df/test-model-71a7df-kserve-6967965559-nd2vz"
Nov 03 22:42:18.022343 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:18.022207    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1018108a-ceed-4fd7-8ad8-23f7478d7cae-model-cache\") pod \"test-model-71a7df-kserve-6967965559-nd2vz\" (UID: \"1018108a-ceed-4fd7-8ad8-23f7478d7cae\") " pod="ai-tenant-e2e-inf-b-71a7df/test-model-71a7df-kserve-6967965559-nd2vz"
Nov 03 22:42:18.022343 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:18.022279    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1018108a-ceed-4fd7-8ad8-23f7478d7cae-home\") pod \"test-model-71a7df-kserve-6967965559-nd2vz\" (UID: \"1018108a-ceed-4fd7-8ad8-23f7478d7cae\") " pod="ai-tenant-e2e-inf-b-71a7df/test-model-71a7df-kserve-6967965559-nd2vz"
Nov 03 22:42:18.022554 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:18.022388    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1018108a-ceed-4fd7-8ad8-23f7478d7cae-tls-certs\") pod \"test-model-71a7df-kserve-6967965559-nd2vz\" (UID: \"1018108a-ceed-4fd7-8ad8-23f7478d7cae\") " pod="ai-tenant-e2e-inf-b-71a7df/test-model-71a7df-kserve-6967965559-nd2vz"
Nov 03 22:42:18.022623 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:18.022572    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1018108a-ceed-4fd7-8ad8-23f7478d7cae-home\") pod \"test-model-71a7df-kserve-6967965559-nd2vz\" (UID: \"1018108a-ceed-4fd7-8ad8-23f7478d7cae\") " pod="ai-tenant-e2e-inf-b-71a7df/test-model-71a7df-kserve-6967965559-nd2vz"
Nov 03 22:42:18.022623 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:18.022594    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1018108a-ceed-4fd7-8ad8-23f7478d7cae-model-cache\") pod \"test-model-71a7df-kserve-6967965559-nd2vz\" (UID: \"1018108a-ceed-4fd7-8ad8-23f7478d7cae\") " pod="ai-tenant-e2e-inf-b-71a7df/test-model-71a7df-kserve-6967965559-nd2vz"
Nov 03 22:42:18.024686 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:18.024657    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1018108a-ceed-4fd7-8ad8-23f7478d7cae-dshm\") pod \"test-model-71a7df-kserve-6967965559-nd2vz\" (UID: \"1018108a-ceed-4fd7-8ad8-23f7478d7cae\") " pod="ai-tenant-e2e-inf-b-71a7df/test-model-71a7df-kserve-6967965559-nd2vz"
Nov 03 22:42:18.025321 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:18.025294    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1018108a-ceed-4fd7-8ad8-23f7478d7cae-tls-certs\") pod \"test-model-71a7df-kserve-6967965559-nd2vz\" (UID: \"1018108a-ceed-4fd7-8ad8-23f7478d7cae\") " pod="ai-tenant-e2e-inf-b-71a7df/test-model-71a7df-kserve-6967965559-nd2vz"
Nov 03 22:42:18.059491 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:18.059454    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swhwv\" (UniqueName: \"kubernetes.io/projected/1018108a-ceed-4fd7-8ad8-23f7478d7cae-kube-api-access-swhwv\") pod \"test-model-71a7df-kserve-6967965559-nd2vz\" (UID: \"1018108a-ceed-4fd7-8ad8-23f7478d7cae\") " pod="ai-tenant-e2e-inf-b-71a7df/test-model-71a7df-kserve-6967965559-nd2vz"
Nov 03 22:42:18.163852 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:18.163819    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-71a7df/test-model-71a7df-kserve-6967965559-nd2vz"
Nov 03 22:42:18.600887 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:18.595765    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-71a7df/test-model-71a7df-kserve-6967965559-nd2vz"]
Nov 03 22:42:18.747590 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:18.747560    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:42:18.936184 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:18.936140    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-71a7df/test-model-71a7df-kserve-6967965559-nd2vz" event={"ID":"1018108a-ceed-4fd7-8ad8-23f7478d7cae","Type":"ContainerStarted","Data":"f984a6dd41f37095f6790345dcbd010e24b408e84dac8664569844455323b1e0"}
Nov 03 22:42:18.936673 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:18.936200    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-71a7df/test-model-71a7df-kserve-6967965559-nd2vz" event={"ID":"1018108a-ceed-4fd7-8ad8-23f7478d7cae","Type":"ContainerStarted","Data":"2c58d8eb8b137c119550ab65409c5e5dafe21dad022859c08e5a11282d556d1c"}
Nov 03 22:42:18.984022 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:18.983962    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-71a7df/test-model-71a7df-kserve-6967965559-nd2vz" podStartSLOduration=1.98394192 podStartE2EDuration="1.98394192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:42:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:42:18.979759373 +0000 UTC m=+1763.005615238" watchObservedRunningTime="2026-11-03 22:42:18.98394192 +0000 UTC m=+1763.009797786"
Nov 03 22:42:19.163974 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:19.163943    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-71a7df/test-model-71a7df-kserve-6967965559-nd2vz"
Nov 03 22:42:20.715753 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:20.715703    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:42:22.529764 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:22.529729    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:42:22.673467 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:22.673428    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:42:22.920554 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:22.920521    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:42:23.415823 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:23.415787    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:42:23.745752 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:23.745671    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:42:24.309795 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:24.309761    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:42:26.006764 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:26.006724    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:42:27.804581 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:27.804491    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:42:28.114286 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:28.114166    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:42:28.164895 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:28.164862    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-71a7df/test-model-71a7df-kserve-6967965559-nd2vz"
Nov 03 22:42:28.177791 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:28.177767    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-71a7df/test-model-71a7df-kserve-6967965559-nd2vz"
Nov 03 22:42:28.296986 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:28.296956    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:42:29.029433 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:29.029404    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-71a7df/test-model-71a7df-kserve-6967965559-nd2vz"
Nov 03 22:42:29.528554 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:29.528514    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:42:29.853348 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:29.853210    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:42:29.995434 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:29.995404    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-fe7ca0-54c86d5ff5-xqxp7"]
Nov 03 22:42:30.000472 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:30.000441    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fe7ca0-54c86d5ff5-xqxp7"
Nov 03 22:42:30.037084 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:30.037047    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-fe7ca0-54c86d5ff5-xqxp7"]
Nov 03 22:42:30.042349 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:30.042319    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:42:30.077658 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:30.077624    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-fe7ca0-dockercfg-j4v4m\""
Nov 03 22:42:30.078186 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:30.078160    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-fe7ca0\""
Nov 03 22:42:30.121008 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:30.120918    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a8675c0-7b21-40aa-bedd-217306c10ade-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-fe7ca0-54c86d5ff5-xqxp7\" (UID: \"8a8675c0-7b21-40aa-bedd-217306c10ade\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fe7ca0-54c86d5ff5-xqxp7"
Nov 03 22:42:30.144298 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:30.124194    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5nhz\" (UniqueName: \"kubernetes.io/projected/8a8675c0-7b21-40aa-bedd-217306c10ade-kube-api-access-k5nhz\") pod \"payload-pre-processing-e2e-inf-b-fe7ca0-54c86d5ff5-xqxp7\" (UID: \"8a8675c0-7b21-40aa-bedd-217306c10ade\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fe7ca0-54c86d5ff5-xqxp7"
Nov 03 22:42:30.267745 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:30.267683    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a8675c0-7b21-40aa-bedd-217306c10ade-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-fe7ca0-54c86d5ff5-xqxp7\" (UID: \"8a8675c0-7b21-40aa-bedd-217306c10ade\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fe7ca0-54c86d5ff5-xqxp7"
Nov 03 22:42:30.267948 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:30.267786    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5nhz\" (UniqueName: \"kubernetes.io/projected/8a8675c0-7b21-40aa-bedd-217306c10ade-kube-api-access-k5nhz\") pod \"payload-pre-processing-e2e-inf-b-fe7ca0-54c86d5ff5-xqxp7\" (UID: \"8a8675c0-7b21-40aa-bedd-217306c10ade\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fe7ca0-54c86d5ff5-xqxp7"
Nov 03 22:42:30.268746 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:30.268717    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a8675c0-7b21-40aa-bedd-217306c10ade-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-fe7ca0-54c86d5ff5-xqxp7\" (UID: \"8a8675c0-7b21-40aa-bedd-217306c10ade\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fe7ca0-54c86d5ff5-xqxp7"
Nov 03 22:42:30.291667 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:30.291631    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5nhz\" (UniqueName: \"kubernetes.io/projected/8a8675c0-7b21-40aa-bedd-217306c10ade-kube-api-access-k5nhz\") pod \"payload-pre-processing-e2e-inf-b-fe7ca0-54c86d5ff5-xqxp7\" (UID: \"8a8675c0-7b21-40aa-bedd-217306c10ade\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fe7ca0-54c86d5ff5-xqxp7"
Nov 03 22:42:30.316887 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:30.316854    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fe7ca0-54c86d5ff5-xqxp7"
Nov 03 22:42:30.629975 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:30.629907    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-fe7ca0-54c86d5ff5-xqxp7"]
Nov 03 22:42:30.630256 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:42:30.630194    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a8675c0_7b21_40aa_bedd_217306c10ade.slice/crio-3d2bb6da7d693e96519672cb4a71a7c4bf62595fcd48c17c82e8b82aa1f871a1 WatchSource:0}: Error finding container 3d2bb6da7d693e96519672cb4a71a7c4bf62595fcd48c17c82e8b82aa1f871a1: Status 404 returned error can't find the container with id 3d2bb6da7d693e96519672cb4a71a7c4bf62595fcd48c17c82e8b82aa1f871a1
Nov 03 22:42:31.034357 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:31.034323    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fe7ca0-54c86d5ff5-xqxp7" event={"ID":"8a8675c0-7b21-40aa-bedd-217306c10ade","Type":"ContainerStarted","Data":"2cd494ca48a81e90c7c396807ff8e9fca37599e3afa776a819585eae82e0c685"}
Nov 03 22:42:31.034556 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:31.034377    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fe7ca0-54c86d5ff5-xqxp7" event={"ID":"8a8675c0-7b21-40aa-bedd-217306c10ade","Type":"ContainerStarted","Data":"3d2bb6da7d693e96519672cb4a71a7c4bf62595fcd48c17c82e8b82aa1f871a1"}
Nov 03 22:42:31.034556 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:31.034463    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fe7ca0-54c86d5ff5-xqxp7"
Nov 03 22:42:31.081861 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:31.081798    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fe7ca0-54c86d5ff5-xqxp7" podStartSLOduration=2.081780571 podStartE2EDuration="2.081780571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:42:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:42:31.078602242 +0000 UTC m=+1775.104458115" watchObservedRunningTime="2026-11-03 22:42:31.081780571 +0000 UTC m=+1775.107636436"
Nov 03 22:42:31.261391 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:42:31.261349    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:42:31.447335 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:31.447296    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:42:32.864337 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:32.864299    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:42:33.018744 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:33.018697    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:42:33.189357 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:33.189315    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:42:34.713768 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:34.713736    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:42:36.197505 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:36.197469    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:42:37.893237 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:37.893016    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:42:39.419949 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:39.419913    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:42:40.891267 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:40.891200    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:42:42.045270 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:42.045208    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fe7ca0-54c86d5ff5-xqxp7"
Nov 03 22:42:42.664622 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:42.664585    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:42:43.509090 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:43.509045    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:42:43.691881 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:43.691849    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:42:44.351482 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:44.351439    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:42:45.850450 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:45.850416    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:42:46.018244 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:46.017674    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:42:47.555948 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:47.555910    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:42:47.717386 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:47.717331    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:42:49.253466 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:49.253430    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:42:51.217380 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:51.217349    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:42:52.822140 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:52.822103    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:42:52.977130 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:52.977091    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:42:54.659486 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:54.659426    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:42:55.756811 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:55.756775    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:42:56.201533 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:56.201495    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:42:56.427054 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:56.427019    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:42:56.654289 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:56.652597    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-963936-openshift-default-68b65f857c-phdvv"]
Nov 03 22:42:56.659905 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:56.659876    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-963936-openshift-default-68b65f857c-phdvv"
Nov 03 22:42:56.695284 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:56.695245    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-963936-openshift-default-68b65f857c-phdvv"]
Nov 03 22:42:56.700152 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:56.700125    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-963936-openshift-default-dockercfg-qwdqd\""
Nov 03 22:42:56.707416 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:56.707089    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mx5q8_ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea/ovn-acl-logging/0.log"
Nov 03 22:42:56.719039 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:56.719011    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mx5q8_ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea/ovn-acl-logging/0.log"
Nov 03 22:42:56.757486 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:56.757423    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsnx9\" (UniqueName: \"kubernetes.io/projected/2cbef074-f335-41d5-a69c-b84a958e87ee-kube-api-access-nsnx9\") pod \"e2e-inf-b-963936-openshift-default-68b65f857c-phdvv\" (UID: \"2cbef074-f335-41d5-a69c-b84a958e87ee\") " pod="openshift-ingress/e2e-inf-b-963936-openshift-default-68b65f857c-phdvv"
Nov 03 22:42:56.758014 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:56.757499    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cbef074-f335-41d5-a69c-b84a958e87ee-istio-envoy\") pod \"e2e-inf-b-963936-openshift-default-68b65f857c-phdvv\" (UID: \"2cbef074-f335-41d5-a69c-b84a958e87ee\") " pod="openshift-ingress/e2e-inf-b-963936-openshift-default-68b65f857c-phdvv"
Nov 03 22:42:56.758014 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:56.757623    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cbef074-f335-41d5-a69c-b84a958e87ee-istio-data\") pod \"e2e-inf-b-963936-openshift-default-68b65f857c-phdvv\" (UID: \"2cbef074-f335-41d5-a69c-b84a958e87ee\") " pod="openshift-ingress/e2e-inf-b-963936-openshift-default-68b65f857c-phdvv"
Nov 03 22:42:56.758014 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:56.757746    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cbef074-f335-41d5-a69c-b84a958e87ee-workload-socket\") pod \"e2e-inf-b-963936-openshift-default-68b65f857c-phdvv\" (UID: \"2cbef074-f335-41d5-a69c-b84a958e87ee\") " pod="openshift-ingress/e2e-inf-b-963936-openshift-default-68b65f857c-phdvv"
Nov 03 22:42:56.758014 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:56.757856    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cbef074-f335-41d5-a69c-b84a958e87ee-credential-socket\") pod \"e2e-inf-b-963936-openshift-default-68b65f857c-phdvv\" (UID: \"2cbef074-f335-41d5-a69c-b84a958e87ee\") " pod="openshift-ingress/e2e-inf-b-963936-openshift-default-68b65f857c-phdvv"
Nov 03 22:42:56.758014 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:56.757943    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cbef074-f335-41d5-a69c-b84a958e87ee-istio-podinfo\") pod \"e2e-inf-b-963936-openshift-default-68b65f857c-phdvv\" (UID: \"2cbef074-f335-41d5-a69c-b84a958e87ee\") " pod="openshift-ingress/e2e-inf-b-963936-openshift-default-68b65f857c-phdvv"
Nov 03 22:42:56.758383 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:56.758054    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cbef074-f335-41d5-a69c-b84a958e87ee-istio-token\") pod \"e2e-inf-b-963936-openshift-default-68b65f857c-phdvv\" (UID: \"2cbef074-f335-41d5-a69c-b84a958e87ee\") " pod="openshift-ingress/e2e-inf-b-963936-openshift-default-68b65f857c-phdvv"
Nov 03 22:42:56.758383 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:56.758112    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cbef074-f335-41d5-a69c-b84a958e87ee-istiod-ca-cert\") pod \"e2e-inf-b-963936-openshift-default-68b65f857c-phdvv\" (UID: \"2cbef074-f335-41d5-a69c-b84a958e87ee\") " pod="openshift-ingress/e2e-inf-b-963936-openshift-default-68b65f857c-phdvv"
Nov 03 22:42:56.758383 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:56.758198    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cbef074-f335-41d5-a69c-b84a958e87ee-workload-certs\") pod \"e2e-inf-b-963936-openshift-default-68b65f857c-phdvv\" (UID: \"2cbef074-f335-41d5-a69c-b84a958e87ee\") " pod="openshift-ingress/e2e-inf-b-963936-openshift-default-68b65f857c-phdvv"
Nov 03 22:42:56.776301 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:56.774573    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:42:56.891345 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:56.891292    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cbef074-f335-41d5-a69c-b84a958e87ee-workload-certs\") pod \"e2e-inf-b-963936-openshift-default-68b65f857c-phdvv\" (UID: \"2cbef074-f335-41d5-a69c-b84a958e87ee\") " pod="openshift-ingress/e2e-inf-b-963936-openshift-default-68b65f857c-phdvv"
Nov 03 22:42:56.892139 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:56.892113    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsnx9\" (UniqueName: \"kubernetes.io/projected/2cbef074-f335-41d5-a69c-b84a958e87ee-kube-api-access-nsnx9\") pod \"e2e-inf-b-963936-openshift-default-68b65f857c-phdvv\" (UID: \"2cbef074-f335-41d5-a69c-b84a958e87ee\") " pod="openshift-ingress/e2e-inf-b-963936-openshift-default-68b65f857c-phdvv"
Nov 03 22:42:56.892344 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:56.892321    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cbef074-f335-41d5-a69c-b84a958e87ee-istio-envoy\") pod \"e2e-inf-b-963936-openshift-default-68b65f857c-phdvv\" (UID: \"2cbef074-f335-41d5-a69c-b84a958e87ee\") " pod="openshift-ingress/e2e-inf-b-963936-openshift-default-68b65f857c-phdvv"
Nov 03 22:42:56.900268 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:56.891974    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cbef074-f335-41d5-a69c-b84a958e87ee-workload-certs\") pod \"e2e-inf-b-963936-openshift-default-68b65f857c-phdvv\" (UID: \"2cbef074-f335-41d5-a69c-b84a958e87ee\") " pod="openshift-ingress/e2e-inf-b-963936-openshift-default-68b65f857c-phdvv"
Nov 03 22:42:56.904239 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:56.901455    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cbef074-f335-41d5-a69c-b84a958e87ee-istio-data\") pod \"e2e-inf-b-963936-openshift-default-68b65f857c-phdvv\" (UID: \"2cbef074-f335-41d5-a69c-b84a958e87ee\") " pod="openshift-ingress/e2e-inf-b-963936-openshift-default-68b65f857c-phdvv"
Nov 03 22:42:56.904239 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:56.901629    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cbef074-f335-41d5-a69c-b84a958e87ee-workload-socket\") pod \"e2e-inf-b-963936-openshift-default-68b65f857c-phdvv\" (UID: \"2cbef074-f335-41d5-a69c-b84a958e87ee\") " pod="openshift-ingress/e2e-inf-b-963936-openshift-default-68b65f857c-phdvv"
Nov 03 22:42:56.904239 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:56.901738    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cbef074-f335-41d5-a69c-b84a958e87ee-credential-socket\") pod \"e2e-inf-b-963936-openshift-default-68b65f857c-phdvv\" (UID: \"2cbef074-f335-41d5-a69c-b84a958e87ee\") " pod="openshift-ingress/e2e-inf-b-963936-openshift-default-68b65f857c-phdvv"
Nov 03 22:42:56.904239 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:56.901844    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cbef074-f335-41d5-a69c-b84a958e87ee-istio-podinfo\") pod \"e2e-inf-b-963936-openshift-default-68b65f857c-phdvv\" (UID: \"2cbef074-f335-41d5-a69c-b84a958e87ee\") " pod="openshift-ingress/e2e-inf-b-963936-openshift-default-68b65f857c-phdvv"
Nov 03 22:42:56.904239 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:56.901944    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cbef074-f335-41d5-a69c-b84a958e87ee-istio-token\") pod \"e2e-inf-b-963936-openshift-default-68b65f857c-phdvv\" (UID: \"2cbef074-f335-41d5-a69c-b84a958e87ee\") " pod="openshift-ingress/e2e-inf-b-963936-openshift-default-68b65f857c-phdvv"
Nov 03 22:42:56.904239 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:56.902001    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cbef074-f335-41d5-a69c-b84a958e87ee-istiod-ca-cert\") pod \"e2e-inf-b-963936-openshift-default-68b65f857c-phdvv\" (UID: \"2cbef074-f335-41d5-a69c-b84a958e87ee\") " pod="openshift-ingress/e2e-inf-b-963936-openshift-default-68b65f857c-phdvv"
Nov 03 22:42:56.904239 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:56.903016    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cbef074-f335-41d5-a69c-b84a958e87ee-istiod-ca-cert\") pod \"e2e-inf-b-963936-openshift-default-68b65f857c-phdvv\" (UID: \"2cbef074-f335-41d5-a69c-b84a958e87ee\") " pod="openshift-ingress/e2e-inf-b-963936-openshift-default-68b65f857c-phdvv"
Nov 03 22:42:56.904702 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:56.904304    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cbef074-f335-41d5-a69c-b84a958e87ee-credential-socket\") pod \"e2e-inf-b-963936-openshift-default-68b65f857c-phdvv\" (UID: \"2cbef074-f335-41d5-a69c-b84a958e87ee\") " pod="openshift-ingress/e2e-inf-b-963936-openshift-default-68b65f857c-phdvv"
Nov 03 22:42:56.904702 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:56.904556    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cbef074-f335-41d5-a69c-b84a958e87ee-istio-data\") pod \"e2e-inf-b-963936-openshift-default-68b65f857c-phdvv\" (UID: \"2cbef074-f335-41d5-a69c-b84a958e87ee\") " pod="openshift-ingress/e2e-inf-b-963936-openshift-default-68b65f857c-phdvv"
Nov 03 22:42:56.904820 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:56.904763    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cbef074-f335-41d5-a69c-b84a958e87ee-workload-socket\") pod \"e2e-inf-b-963936-openshift-default-68b65f857c-phdvv\" (UID: \"2cbef074-f335-41d5-a69c-b84a958e87ee\") " pod="openshift-ingress/e2e-inf-b-963936-openshift-default-68b65f857c-phdvv"
Nov 03 22:42:56.929360 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:56.929327    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cbef074-f335-41d5-a69c-b84a958e87ee-istio-podinfo\") pod \"e2e-inf-b-963936-openshift-default-68b65f857c-phdvv\" (UID: \"2cbef074-f335-41d5-a69c-b84a958e87ee\") " pod="openshift-ingress/e2e-inf-b-963936-openshift-default-68b65f857c-phdvv"
Nov 03 22:42:56.930329 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:56.930304    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cbef074-f335-41d5-a69c-b84a958e87ee-istio-envoy\") pod \"e2e-inf-b-963936-openshift-default-68b65f857c-phdvv\" (UID: \"2cbef074-f335-41d5-a69c-b84a958e87ee\") " pod="openshift-ingress/e2e-inf-b-963936-openshift-default-68b65f857c-phdvv"
Nov 03 22:42:56.932802 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:56.932758    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:42:56.954065 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:56.954029    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsnx9\" (UniqueName: \"kubernetes.io/projected/2cbef074-f335-41d5-a69c-b84a958e87ee-kube-api-access-nsnx9\") pod \"e2e-inf-b-963936-openshift-default-68b65f857c-phdvv\" (UID: \"2cbef074-f335-41d5-a69c-b84a958e87ee\") " pod="openshift-ingress/e2e-inf-b-963936-openshift-default-68b65f857c-phdvv"
Nov 03 22:42:56.954775 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:56.954752    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cbef074-f335-41d5-a69c-b84a958e87ee-istio-token\") pod \"e2e-inf-b-963936-openshift-default-68b65f857c-phdvv\" (UID: \"2cbef074-f335-41d5-a69c-b84a958e87ee\") " pod="openshift-ingress/e2e-inf-b-963936-openshift-default-68b65f857c-phdvv"
Nov 03 22:42:57.021133 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:57.021101    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-963936-openshift-default-68b65f857c-phdvv"
Nov 03 22:42:57.461605 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:57.461571    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-963936-openshift-default-68b65f857c-phdvv"]
Nov 03 22:42:57.466108 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:42:57.465272    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cbef074_f335_41d5_a69c_b84a958e87ee.slice/crio-9dd6dbf3831aa5ea22e88248af2bbdb00b85d345018f77597e9fbd4fbec65cd8 WatchSource:0}: Error finding container 9dd6dbf3831aa5ea22e88248af2bbdb00b85d345018f77597e9fbd4fbec65cd8: Status 404 returned error can't find the container with id 9dd6dbf3831aa5ea22e88248af2bbdb00b85d345018f77597e9fbd4fbec65cd8
Nov 03 22:42:57.469366 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:57.469330    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:42:57.469472 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:57.469399    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:42:57.469472 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:57.469428    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:42:58.116356 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:58.116294    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:42:58.301736 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:58.301682    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-963936-openshift-default-68b65f857c-phdvv" event={"ID":"2cbef074-f335-41d5-a69c-b84a958e87ee","Type":"ContainerStarted","Data":"12bd9f676d34dd68d6e917b86ac592cd27ca4690bd71c1ecf29d513985017952"}
Nov 03 22:42:58.301923 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:58.301751    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-963936-openshift-default-68b65f857c-phdvv" event={"ID":"2cbef074-f335-41d5-a69c-b84a958e87ee","Type":"ContainerStarted","Data":"9dd6dbf3831aa5ea22e88248af2bbdb00b85d345018f77597e9fbd4fbec65cd8"}
Nov 03 22:42:58.360177 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:58.360115    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-963936-openshift-default-68b65f857c-phdvv" podStartSLOduration=2.360092813 podStartE2EDuration="2.360092813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:42:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:42:58.357240976 +0000 UTC m=+1802.383096828" watchObservedRunningTime="2026-11-03 22:42:58.360092813 +0000 UTC m=+1802.385948676"
Nov 03 22:42:59.023850 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:59.023803    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-963936-openshift-default-68b65f857c-phdvv"
Nov 03 22:42:59.032140 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:59.032110    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-963936-openshift-default-68b65f857c-phdvv"
Nov 03 22:42:59.310667 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:59.310592    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-963936-openshift-default-68b65f857c-phdvv"
Nov 03 22:42:59.311710 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:59.311688    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-963936-openshift-default-68b65f857c-phdvv"
Nov 03 22:42:59.877246 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:42:59.873480    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:43:01.115191 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:01.115151    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:43:01.260704 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:43:01.260660    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:43:01.327351 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:01.327207    4431 generic.go:356] "Generic (PLEG): container finished" podID="51131709-120f-4b00-a6ec-34e879ac168b" containerID="33077418fd8a5ad61a910ab6c88db15081db2621f5b0591440c405b460d30f02" exitCode=0
Nov 03 22:43:01.327535 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:01.327339    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq" event={"ID":"51131709-120f-4b00-a6ec-34e879ac168b","Type":"ContainerDied","Data":"33077418fd8a5ad61a910ab6c88db15081db2621f5b0591440c405b460d30f02"}
Nov 03 22:43:01.328126 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:01.328099    4431 scope.go:122] "RemoveContainer" containerID="33077418fd8a5ad61a910ab6c88db15081db2621f5b0591440c405b460d30f02"
Nov 03 22:43:01.329660 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:01.329632    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:43:01.330130 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:01.330101    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:43:01.330267 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:01.330190    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:43:01.330267 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:01.330259    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:43:01.649266 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:01.649168    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:43:01.816137 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:01.814239    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:43:02.336514 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:02.336481    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq" event={"ID":"51131709-120f-4b00-a6ec-34e879ac168b","Type":"ContainerStarted","Data":"254a92a8fc5eb9d34807583d4d95b4862e8f0d42259922f29746d533d66b4d49"}
Nov 03 22:43:02.450542 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:02.439427    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq"
Nov 03 22:43:02.462324 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:02.457823    4431 patch_prober.go:29] interesting pod/e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Nov 03 22:43:02.462324 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:02.457897    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq" podUID="51131709-120f-4b00-a6ec-34e879ac168b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Nov 03 22:43:02.780282 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:02.779465    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:43:02.936885 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:02.936853    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:43:03.450241 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:03.442043    4431 patch_prober.go:29] interesting pod/e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Nov 03 22:43:03.450241 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:03.442098    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq" podUID="51131709-120f-4b00-a6ec-34e879ac168b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Nov 03 22:43:03.572243 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:03.569245    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:43:04.462239 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:04.462043    4431 patch_prober.go:29] interesting pod/e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Nov 03 22:43:04.462239 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:04.462113    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq" podUID="51131709-120f-4b00-a6ec-34e879ac168b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Nov 03 22:43:04.708821 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:04.708791    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:43:05.166117 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:05.166086    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:43:05.399887 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:05.399853    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:43:05.469760 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:05.469654    4431 patch_prober.go:29] interesting pod/e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Nov 03 22:43:05.470320 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:05.469785    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq" podUID="51131709-120f-4b00-a6ec-34e879ac168b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Nov 03 22:43:05.688949 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:05.688896    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:43:05.848291 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:05.846116    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:43:06.443237 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:06.442686    4431 patch_prober.go:29] interesting pod/e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Nov 03 22:43:06.443237 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:06.442749    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq" podUID="51131709-120f-4b00-a6ec-34e879ac168b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Nov 03 22:43:07.017171 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:07.017136    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:43:07.115253 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:07.115206    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq"
Nov 03 22:43:07.442665 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:07.442625    4431 patch_prober.go:29] interesting pod/e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Nov 03 22:43:07.442972 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:07.442939    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq" podUID="51131709-120f-4b00-a6ec-34e879ac168b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Nov 03 22:43:08.444167 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:08.444137    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq"
Nov 03 22:43:08.445442 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:08.445420    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq"
Nov 03 22:43:08.664845 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:08.664805    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:43:09.548496 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:09.548457    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:43:09.698927 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:09.698891    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:43:10.491938 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:10.491902    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:43:11.175841 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:11.175805    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:43:11.289871 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:11.289838    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-124490-7cbff46899-7c5p6"]
Nov 03 22:43:11.296411 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:11.296381    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-124490-7cbff46899-7c5p6"
Nov 03 22:43:11.317199 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:11.317170    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-124490-7cbff46899-7c5p6"]
Nov 03 22:43:11.321676 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:11.321646    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-124490-dockercfg-xwlw8\""
Nov 03 22:43:11.323847 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:11.323825    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:43:11.385185 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:11.385139    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-124490\""
Nov 03 22:43:11.394357 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:11.394317    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7vww\" (UniqueName: \"kubernetes.io/projected/fffa2baf-eaca-4459-b8c4-63ded79ac8ec-kube-api-access-m7vww\") pod \"payload-pre-processing-e2e-inf-b-124490-7cbff46899-7c5p6\" (UID: \"fffa2baf-eaca-4459-b8c4-63ded79ac8ec\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-124490-7cbff46899-7c5p6"
Nov 03 22:43:11.394596 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:11.394442    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fffa2baf-eaca-4459-b8c4-63ded79ac8ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-124490-7cbff46899-7c5p6\" (UID: \"fffa2baf-eaca-4459-b8c4-63ded79ac8ec\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-124490-7cbff46899-7c5p6"
Nov 03 22:43:11.514286 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:11.511294    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7vww\" (UniqueName: \"kubernetes.io/projected/fffa2baf-eaca-4459-b8c4-63ded79ac8ec-kube-api-access-m7vww\") pod \"payload-pre-processing-e2e-inf-b-124490-7cbff46899-7c5p6\" (UID: \"fffa2baf-eaca-4459-b8c4-63ded79ac8ec\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-124490-7cbff46899-7c5p6"
Nov 03 22:43:11.514286 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:11.511436    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fffa2baf-eaca-4459-b8c4-63ded79ac8ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-124490-7cbff46899-7c5p6\" (UID: \"fffa2baf-eaca-4459-b8c4-63ded79ac8ec\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-124490-7cbff46899-7c5p6"
Nov 03 22:43:11.514286 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:11.512522    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fffa2baf-eaca-4459-b8c4-63ded79ac8ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-124490-7cbff46899-7c5p6\" (UID: \"fffa2baf-eaca-4459-b8c4-63ded79ac8ec\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-124490-7cbff46899-7c5p6"
Nov 03 22:43:11.544727 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:11.544664    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7vww\" (UniqueName: \"kubernetes.io/projected/fffa2baf-eaca-4459-b8c4-63ded79ac8ec-kube-api-access-m7vww\") pod \"payload-pre-processing-e2e-inf-b-124490-7cbff46899-7c5p6\" (UID: \"fffa2baf-eaca-4459-b8c4-63ded79ac8ec\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-124490-7cbff46899-7c5p6"
Nov 03 22:43:11.620130 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:11.620098    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-124490-7cbff46899-7c5p6"
Nov 03 22:43:11.934074 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:11.934044    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-124490-7cbff46899-7c5p6"]
Nov 03 22:43:11.935391 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:43:11.935365    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfffa2baf_eaca_4459_b8c4_63ded79ac8ec.slice/crio-270f9230be844fa7a769d412bfc477b36164cbab679a61770be3df12de19b26e WatchSource:0}: Error finding container 270f9230be844fa7a769d412bfc477b36164cbab679a61770be3df12de19b26e: Status 404 returned error can't find the container with id 270f9230be844fa7a769d412bfc477b36164cbab679a61770be3df12de19b26e
Nov 03 22:43:12.231080 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:12.230997    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:43:12.500564 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:12.500475    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-124490-7cbff46899-7c5p6" event={"ID":"fffa2baf-eaca-4459-b8c4-63ded79ac8ec","Type":"ContainerStarted","Data":"f1bf17e8fb23a68a85b76df4d96f59f886a0b5f682a5efe587ab02b34eaba549"}
Nov 03 22:43:12.500564 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:12.500527    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-124490-7cbff46899-7c5p6" event={"ID":"fffa2baf-eaca-4459-b8c4-63ded79ac8ec","Type":"ContainerStarted","Data":"270f9230be844fa7a769d412bfc477b36164cbab679a61770be3df12de19b26e"}
Nov 03 22:43:12.500762 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:12.500682    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-124490-7cbff46899-7c5p6"
Nov 03 22:43:12.545583 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:12.545519    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-124490-7cbff46899-7c5p6" podStartSLOduration=1.545502175 podStartE2EDuration="1.545502175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:43:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:43:12.542401346 +0000 UTC m=+1816.568257207" watchObservedRunningTime="2026-11-03 22:43:12.545502175 +0000 UTC m=+1816.571358060"
Nov 03 22:43:13.770336 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:13.770298    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:43:13.912391 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:13.912343    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:43:14.247286 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:14.247254    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:43:15.524976 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:15.524941    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:43:17.152872 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:17.152794    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:43:17.225912 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:17.225876    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5"]
Nov 03 22:43:17.226176 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:17.226148    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5" podUID="bb2c0928-23fa-462f-baa4-4bd63dab4d62" containerName="istio-proxy" containerID="cri-o://bd14f1cdfdad496f626af92146563f497107c9b8015311eca462a239d0cc1d2d" gracePeriod=30
Nov 03 22:43:18.101852 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:18.101823    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-sf6vl"]
Nov 03 22:43:18.106946 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:18.106926    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-sf6vl"
Nov 03 22:43:18.124634 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:18.124605    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-0a7065-openshift-default-dockercfg-bs5ch\""
Nov 03 22:43:18.143747 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:18.143710    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-sf6vl"]
Nov 03 22:43:18.158966 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:18.158936    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c772f6e-d9c8-4dbe-aff9-b6b6026032ae-workload-socket\") pod \"e2e-inf-b-0a7065-openshift-default-56696cf555-sf6vl\" (UID: \"2c772f6e-d9c8-4dbe-aff9-b6b6026032ae\") " pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-sf6vl"
Nov 03 22:43:18.159459 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:18.159276    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c772f6e-d9c8-4dbe-aff9-b6b6026032ae-credential-socket\") pod \"e2e-inf-b-0a7065-openshift-default-56696cf555-sf6vl\" (UID: \"2c772f6e-d9c8-4dbe-aff9-b6b6026032ae\") " pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-sf6vl"
Nov 03 22:43:18.159459 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:18.159317    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c772f6e-d9c8-4dbe-aff9-b6b6026032ae-workload-certs\") pod \"e2e-inf-b-0a7065-openshift-default-56696cf555-sf6vl\" (UID: \"2c772f6e-d9c8-4dbe-aff9-b6b6026032ae\") " pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-sf6vl"
Nov 03 22:43:18.159459 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:18.159393    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c772f6e-d9c8-4dbe-aff9-b6b6026032ae-istio-data\") pod \"e2e-inf-b-0a7065-openshift-default-56696cf555-sf6vl\" (UID: \"2c772f6e-d9c8-4dbe-aff9-b6b6026032ae\") " pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-sf6vl"
Nov 03 22:43:18.159459 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:18.159431    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c772f6e-d9c8-4dbe-aff9-b6b6026032ae-istiod-ca-cert\") pod \"e2e-inf-b-0a7065-openshift-default-56696cf555-sf6vl\" (UID: \"2c772f6e-d9c8-4dbe-aff9-b6b6026032ae\") " pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-sf6vl"
Nov 03 22:43:18.159691 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:18.159562    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c772f6e-d9c8-4dbe-aff9-b6b6026032ae-istio-envoy\") pod \"e2e-inf-b-0a7065-openshift-default-56696cf555-sf6vl\" (UID: \"2c772f6e-d9c8-4dbe-aff9-b6b6026032ae\") " pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-sf6vl"
Nov 03 22:43:18.159691 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:18.159608    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c772f6e-d9c8-4dbe-aff9-b6b6026032ae-istio-token\") pod \"e2e-inf-b-0a7065-openshift-default-56696cf555-sf6vl\" (UID: \"2c772f6e-d9c8-4dbe-aff9-b6b6026032ae\") " pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-sf6vl"
Nov 03 22:43:18.159691 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:18.159629    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c772f6e-d9c8-4dbe-aff9-b6b6026032ae-istio-podinfo\") pod \"e2e-inf-b-0a7065-openshift-default-56696cf555-sf6vl\" (UID: \"2c772f6e-d9c8-4dbe-aff9-b6b6026032ae\") " pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-sf6vl"
Nov 03 22:43:18.159691 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:18.159648    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79lq8\" (UniqueName: \"kubernetes.io/projected/2c772f6e-d9c8-4dbe-aff9-b6b6026032ae-kube-api-access-79lq8\") pod \"e2e-inf-b-0a7065-openshift-default-56696cf555-sf6vl\" (UID: \"2c772f6e-d9c8-4dbe-aff9-b6b6026032ae\") " pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-sf6vl"
Nov 03 22:43:18.260829 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:18.260794    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c772f6e-d9c8-4dbe-aff9-b6b6026032ae-credential-socket\") pod \"e2e-inf-b-0a7065-openshift-default-56696cf555-sf6vl\" (UID: \"2c772f6e-d9c8-4dbe-aff9-b6b6026032ae\") " pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-sf6vl"
Nov 03 22:43:18.261000 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:18.260845    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c772f6e-d9c8-4dbe-aff9-b6b6026032ae-workload-certs\") pod \"e2e-inf-b-0a7065-openshift-default-56696cf555-sf6vl\" (UID: \"2c772f6e-d9c8-4dbe-aff9-b6b6026032ae\") " pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-sf6vl"
Nov 03 22:43:18.261000 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:18.260885    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c772f6e-d9c8-4dbe-aff9-b6b6026032ae-istio-data\") pod \"e2e-inf-b-0a7065-openshift-default-56696cf555-sf6vl\" (UID: \"2c772f6e-d9c8-4dbe-aff9-b6b6026032ae\") " pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-sf6vl"
Nov 03 22:43:18.261000 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:18.260943    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c772f6e-d9c8-4dbe-aff9-b6b6026032ae-istiod-ca-cert\") pod \"e2e-inf-b-0a7065-openshift-default-56696cf555-sf6vl\" (UID: \"2c772f6e-d9c8-4dbe-aff9-b6b6026032ae\") " pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-sf6vl"
Nov 03 22:43:18.261171 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:18.261012    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c772f6e-d9c8-4dbe-aff9-b6b6026032ae-istio-envoy\") pod \"e2e-inf-b-0a7065-openshift-default-56696cf555-sf6vl\" (UID: \"2c772f6e-d9c8-4dbe-aff9-b6b6026032ae\") " pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-sf6vl"
Nov 03 22:43:18.261171 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:18.261085    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c772f6e-d9c8-4dbe-aff9-b6b6026032ae-istio-token\") pod \"e2e-inf-b-0a7065-openshift-default-56696cf555-sf6vl\" (UID: \"2c772f6e-d9c8-4dbe-aff9-b6b6026032ae\") " pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-sf6vl"
Nov 03 22:43:18.261171 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:18.261118    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c772f6e-d9c8-4dbe-aff9-b6b6026032ae-istio-podinfo\") pod \"e2e-inf-b-0a7065-openshift-default-56696cf555-sf6vl\" (UID: \"2c772f6e-d9c8-4dbe-aff9-b6b6026032ae\") " pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-sf6vl"
Nov 03 22:43:18.261171 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:18.261148    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79lq8\" (UniqueName: \"kubernetes.io/projected/2c772f6e-d9c8-4dbe-aff9-b6b6026032ae-kube-api-access-79lq8\") pod \"e2e-inf-b-0a7065-openshift-default-56696cf555-sf6vl\" (UID: \"2c772f6e-d9c8-4dbe-aff9-b6b6026032ae\") " pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-sf6vl"
Nov 03 22:43:18.261448 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:18.261209    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c772f6e-d9c8-4dbe-aff9-b6b6026032ae-credential-socket\") pod \"e2e-inf-b-0a7065-openshift-default-56696cf555-sf6vl\" (UID: \"2c772f6e-d9c8-4dbe-aff9-b6b6026032ae\") " pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-sf6vl"
Nov 03 22:43:18.261448 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:18.261309    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c772f6e-d9c8-4dbe-aff9-b6b6026032ae-workload-socket\") pod \"e2e-inf-b-0a7065-openshift-default-56696cf555-sf6vl\" (UID: \"2c772f6e-d9c8-4dbe-aff9-b6b6026032ae\") " pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-sf6vl"
Nov 03 22:43:18.261578 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:18.261546    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c772f6e-d9c8-4dbe-aff9-b6b6026032ae-workload-certs\") pod \"e2e-inf-b-0a7065-openshift-default-56696cf555-sf6vl\" (UID: \"2c772f6e-d9c8-4dbe-aff9-b6b6026032ae\") " pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-sf6vl"
Nov 03 22:43:18.261646 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:18.261571    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c772f6e-d9c8-4dbe-aff9-b6b6026032ae-istio-data\") pod \"e2e-inf-b-0a7065-openshift-default-56696cf555-sf6vl\" (UID: \"2c772f6e-d9c8-4dbe-aff9-b6b6026032ae\") " pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-sf6vl"
Nov 03 22:43:18.261788 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:18.261767    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c772f6e-d9c8-4dbe-aff9-b6b6026032ae-istiod-ca-cert\") pod \"e2e-inf-b-0a7065-openshift-default-56696cf555-sf6vl\" (UID: \"2c772f6e-d9c8-4dbe-aff9-b6b6026032ae\") " pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-sf6vl"
Nov 03 22:43:18.261849 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:18.261772    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c772f6e-d9c8-4dbe-aff9-b6b6026032ae-workload-socket\") pod \"e2e-inf-b-0a7065-openshift-default-56696cf555-sf6vl\" (UID: \"2c772f6e-d9c8-4dbe-aff9-b6b6026032ae\") " pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-sf6vl"
Nov 03 22:43:18.263559 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:18.263542    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c772f6e-d9c8-4dbe-aff9-b6b6026032ae-istio-envoy\") pod \"e2e-inf-b-0a7065-openshift-default-56696cf555-sf6vl\" (UID: \"2c772f6e-d9c8-4dbe-aff9-b6b6026032ae\") " pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-sf6vl"
Nov 03 22:43:18.263783 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:18.263764    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c772f6e-d9c8-4dbe-aff9-b6b6026032ae-istio-podinfo\") pod \"e2e-inf-b-0a7065-openshift-default-56696cf555-sf6vl\" (UID: \"2c772f6e-d9c8-4dbe-aff9-b6b6026032ae\") " pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-sf6vl"
Nov 03 22:43:18.284827 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:18.284799    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c772f6e-d9c8-4dbe-aff9-b6b6026032ae-istio-token\") pod \"e2e-inf-b-0a7065-openshift-default-56696cf555-sf6vl\" (UID: \"2c772f6e-d9c8-4dbe-aff9-b6b6026032ae\") " pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-sf6vl"
Nov 03 22:43:18.284998 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:18.284914    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79lq8\" (UniqueName: \"kubernetes.io/projected/2c772f6e-d9c8-4dbe-aff9-b6b6026032ae-kube-api-access-79lq8\") pod \"e2e-inf-b-0a7065-openshift-default-56696cf555-sf6vl\" (UID: \"2c772f6e-d9c8-4dbe-aff9-b6b6026032ae\") " pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-sf6vl"
Nov 03 22:43:18.422885 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:18.422848    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-sf6vl"
Nov 03 22:43:18.751790 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:18.751761    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-sf6vl"]
Nov 03 22:43:18.752397 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:43:18.752368    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c772f6e_d9c8_4dbe_aff9_b6b6026032ae.slice/crio-9dfd6453e69d288f1e5b6897a49af83e0dd55ccfa362e7ed83c1c2bd83a378ed WatchSource:0}: Error finding container 9dfd6453e69d288f1e5b6897a49af83e0dd55ccfa362e7ed83c1c2bd83a378ed: Status 404 returned error can't find the container with id 9dfd6453e69d288f1e5b6897a49af83e0dd55ccfa362e7ed83c1c2bd83a378ed
Nov 03 22:43:18.756080 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:18.756037    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:43:18.756173 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:18.756115    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:43:18.756173 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:18.756145    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:43:18.811010 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:18.810919    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:43:19.590563 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:19.590521    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-sf6vl" event={"ID":"2c772f6e-d9c8-4dbe-aff9-b6b6026032ae","Type":"ContainerStarted","Data":"c8b0389c8d6b5eba8941413bf88b058c2e6b46b78ec8864e78f8edea0e504566"}
Nov 03 22:43:19.590984 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:19.590594    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-sf6vl" event={"ID":"2c772f6e-d9c8-4dbe-aff9-b6b6026032ae","Type":"ContainerStarted","Data":"9dfd6453e69d288f1e5b6897a49af83e0dd55ccfa362e7ed83c1c2bd83a378ed"}
Nov 03 22:43:19.636883 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:19.636826    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-sf6vl" podStartSLOduration=1.636808558 podStartE2EDuration="1.636808558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:43:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:43:19.63447643 +0000 UTC m=+1823.660332304" watchObservedRunningTime="2026-11-03 22:43:19.636808558 +0000 UTC m=+1823.662664420"
Nov 03 22:43:20.225576 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:20.225540    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:43:20.364568 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:20.364532    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:43:20.423313 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:20.423271    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-sf6vl"
Nov 03 22:43:20.425201 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:20.425173    4431 patch_prober.go:29] interesting pod/e2e-inf-b-0a7065-openshift-default-56696cf555-sf6vl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Nov 03 22:43:20.425394 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:20.425360    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-sf6vl" podUID="2c772f6e-d9c8-4dbe-aff9-b6b6026032ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Nov 03 22:43:20.506520 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:20.506435    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:43:21.423598 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:21.423563    4431 patch_prober.go:29] interesting pod/e2e-inf-b-0a7065-openshift-default-56696cf555-sf6vl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Nov 03 22:43:21.423994 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:21.423619    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-sf6vl" podUID="2c772f6e-d9c8-4dbe-aff9-b6b6026032ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Nov 03 22:43:22.012945 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:22.012902    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:43:22.110288 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:22.110241    4431 patch_prober.go:29] interesting pod/e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 22:43:22.110476 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:22.110307    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5" podUID="bb2c0928-23fa-462f-baa4-4bd63dab4d62" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 22:43:22.424347 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:22.424262    4431 patch_prober.go:29] interesting pod/e2e-inf-b-0a7065-openshift-default-56696cf555-sf6vl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Nov 03 22:43:22.424347 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:22.424328    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-sf6vl" podUID="2c772f6e-d9c8-4dbe-aff9-b6b6026032ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Nov 03 22:43:22.605858 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:22.605833    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5"
Nov 03 22:43:22.616610 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:22.616582    4431 generic.go:356] "Generic (PLEG): container finished" podID="bb2c0928-23fa-462f-baa4-4bd63dab4d62" containerID="bd14f1cdfdad496f626af92146563f497107c9b8015311eca462a239d0cc1d2d" exitCode=0
Nov 03 22:43:22.616755 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:22.616642    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5"
Nov 03 22:43:22.616755 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:22.616669    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5" event={"ID":"bb2c0928-23fa-462f-baa4-4bd63dab4d62","Type":"ContainerDied","Data":"bd14f1cdfdad496f626af92146563f497107c9b8015311eca462a239d0cc1d2d"}
Nov 03 22:43:22.616755 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:22.616728    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5" event={"ID":"bb2c0928-23fa-462f-baa4-4bd63dab4d62","Type":"ContainerDied","Data":"0a82386da5b90be5b6e1b671a757a3c46daf57cedf1217557f7073a745b17bb7"}
Nov 03 22:43:22.616755 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:22.616753    4431 scope.go:122] "RemoveContainer" containerID="bd14f1cdfdad496f626af92146563f497107c9b8015311eca462a239d0cc1d2d"
Nov 03 22:43:22.627506 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:22.627479    4431 scope.go:122] "RemoveContainer" containerID="bd14f1cdfdad496f626af92146563f497107c9b8015311eca462a239d0cc1d2d"
Nov 03 22:43:22.627786 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:43:22.627766    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd14f1cdfdad496f626af92146563f497107c9b8015311eca462a239d0cc1d2d\": container with ID starting with bd14f1cdfdad496f626af92146563f497107c9b8015311eca462a239d0cc1d2d not found: ID does not exist" containerID="bd14f1cdfdad496f626af92146563f497107c9b8015311eca462a239d0cc1d2d"
Nov 03 22:43:22.627830 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:22.627793    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd14f1cdfdad496f626af92146563f497107c9b8015311eca462a239d0cc1d2d"} err="failed to get container status \"bd14f1cdfdad496f626af92146563f497107c9b8015311eca462a239d0cc1d2d\": rpc error: code = NotFound desc = could not find container \"bd14f1cdfdad496f626af92146563f497107c9b8015311eca462a239d0cc1d2d\": container with ID starting with bd14f1cdfdad496f626af92146563f497107c9b8015311eca462a239d0cc1d2d not found: ID does not exist"
Nov 03 22:43:22.779433 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:22.779359    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb2c0928-23fa-462f-baa4-4bd63dab4d62-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb2c0928-23fa-462f-baa4-4bd63dab4d62-istio-data\") pod \"bb2c0928-23fa-462f-baa4-4bd63dab4d62\" (UID: \"bb2c0928-23fa-462f-baa4-4bd63dab4d62\") "
Nov 03 22:43:22.781333 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:22.781305    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bb2c0928-23fa-462f-baa4-4bd63dab4d62-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb2c0928-23fa-462f-baa4-4bd63dab4d62-istio-podinfo\") pod \"bb2c0928-23fa-462f-baa4-4bd63dab4d62\" (UID: \"bb2c0928-23fa-462f-baa4-4bd63dab4d62\") "
Nov 03 22:43:22.781524 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:22.781509    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb2c0928-23fa-462f-baa4-4bd63dab4d62-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb2c0928-23fa-462f-baa4-4bd63dab4d62-istio-envoy\") pod \"bb2c0928-23fa-462f-baa4-4bd63dab4d62\" (UID: \"bb2c0928-23fa-462f-baa4-4bd63dab4d62\") "
Nov 03 22:43:22.781738 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:22.781704    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb2c0928-23fa-462f-baa4-4bd63dab4d62-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb2c0928-23fa-462f-baa4-4bd63dab4d62-credential-socket\") pod \"bb2c0928-23fa-462f-baa4-4bd63dab4d62\" (UID: \"bb2c0928-23fa-462f-baa4-4bd63dab4d62\") "
Nov 03 22:43:22.782184 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:22.781545    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb2c0928-23fa-462f-baa4-4bd63dab4d62-istio-data" pod "bb2c0928-23fa-462f-baa4-4bd63dab4d62" (UID: "bb2c0928-23fa-462f-baa4-4bd63dab4d62"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:43:22.782335 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:22.782122    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb2c0928-23fa-462f-baa4-4bd63dab4d62-credential-socket" pod "bb2c0928-23fa-462f-baa4-4bd63dab4d62" (UID: "bb2c0928-23fa-462f-baa4-4bd63dab4d62"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:43:22.782973 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:22.782658    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb2c0928-23fa-462f-baa4-4bd63dab4d62-kube-api-access-pk48z\" (UniqueName: \"kubernetes.io/projected/bb2c0928-23fa-462f-baa4-4bd63dab4d62-kube-api-access-pk48z\") pod \"bb2c0928-23fa-462f-baa4-4bd63dab4d62\" (UID: \"bb2c0928-23fa-462f-baa4-4bd63dab4d62\") "
Nov 03 22:43:22.782973 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:22.782719    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb2c0928-23fa-462f-baa4-4bd63dab4d62-istio-token\" (UniqueName: \"kubernetes.io/projected/bb2c0928-23fa-462f-baa4-4bd63dab4d62-istio-token\") pod \"bb2c0928-23fa-462f-baa4-4bd63dab4d62\" (UID: \"bb2c0928-23fa-462f-baa4-4bd63dab4d62\") "
Nov 03 22:43:22.782973 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:22.782834    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb2c0928-23fa-462f-baa4-4bd63dab4d62-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb2c0928-23fa-462f-baa4-4bd63dab4d62-istiod-ca-cert\") pod \"bb2c0928-23fa-462f-baa4-4bd63dab4d62\" (UID: \"bb2c0928-23fa-462f-baa4-4bd63dab4d62\") "
Nov 03 22:43:22.783197 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:22.782974    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb2c0928-23fa-462f-baa4-4bd63dab4d62-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb2c0928-23fa-462f-baa4-4bd63dab4d62-workload-certs\") pod \"bb2c0928-23fa-462f-baa4-4bd63dab4d62\" (UID: \"bb2c0928-23fa-462f-baa4-4bd63dab4d62\") "
Nov 03 22:43:22.783197 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:22.783019    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb2c0928-23fa-462f-baa4-4bd63dab4d62-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb2c0928-23fa-462f-baa4-4bd63dab4d62-workload-socket\") pod \"bb2c0928-23fa-462f-baa4-4bd63dab4d62\" (UID: \"bb2c0928-23fa-462f-baa4-4bd63dab4d62\") "
Nov 03 22:43:22.783950 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:22.783925    4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb2c0928-23fa-462f-baa4-4bd63dab4d62-istio-data\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:43:22.784055 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:22.783956    4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb2c0928-23fa-462f-baa4-4bd63dab4d62-credential-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:43:22.784166 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:22.784119    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb2c0928-23fa-462f-baa4-4bd63dab4d62-workload-certs" pod "bb2c0928-23fa-462f-baa4-4bd63dab4d62" (UID: "bb2c0928-23fa-462f-baa4-4bd63dab4d62"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:43:22.784263 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:22.784188    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb2c0928-23fa-462f-baa4-4bd63dab4d62-workload-socket" pod "bb2c0928-23fa-462f-baa4-4bd63dab4d62" (UID: "bb2c0928-23fa-462f-baa4-4bd63dab4d62"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:43:22.784320 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:22.784254    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb2c0928-23fa-462f-baa4-4bd63dab4d62-istiod-ca-cert" pod "bb2c0928-23fa-462f-baa4-4bd63dab4d62" (UID: "bb2c0928-23fa-462f-baa4-4bd63dab4d62"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:43:22.784775 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:22.784749    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb2c0928-23fa-462f-baa4-4bd63dab4d62-istio-envoy" pod "bb2c0928-23fa-462f-baa4-4bd63dab4d62" (UID: "bb2c0928-23fa-462f-baa4-4bd63dab4d62"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:43:22.785989 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:22.785964    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb2c0928-23fa-462f-baa4-4bd63dab4d62-istio-token" pod "bb2c0928-23fa-462f-baa4-4bd63dab4d62" (UID: "bb2c0928-23fa-462f-baa4-4bd63dab4d62"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:43:22.786297 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:22.786198    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb2c0928-23fa-462f-baa4-4bd63dab4d62-kube-api-access-pk48z" pod "bb2c0928-23fa-462f-baa4-4bd63dab4d62" (UID: "bb2c0928-23fa-462f-baa4-4bd63dab4d62"). InnerVolumeSpecName "kube-api-access-pk48z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:43:22.786613 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:22.786588    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bb2c0928-23fa-462f-baa4-4bd63dab4d62-istio-podinfo" pod "bb2c0928-23fa-462f-baa4-4bd63dab4d62" (UID: "bb2c0928-23fa-462f-baa4-4bd63dab4d62"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 22:43:22.885604 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:22.885525    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pk48z\" (UniqueName: \"kubernetes.io/projected/bb2c0928-23fa-462f-baa4-4bd63dab4d62-kube-api-access-pk48z\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:43:22.885604 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:22.885554    4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb2c0928-23fa-462f-baa4-4bd63dab4d62-istio-token\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:43:22.885604 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:22.885564    4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb2c0928-23fa-462f-baa4-4bd63dab4d62-istiod-ca-cert\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:43:22.885604 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:22.885574    4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb2c0928-23fa-462f-baa4-4bd63dab4d62-workload-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:43:22.885604 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:22.885582    4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb2c0928-23fa-462f-baa4-4bd63dab4d62-workload-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:43:22.885604 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:22.885591    4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb2c0928-23fa-462f-baa4-4bd63dab4d62-istio-podinfo\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:43:22.885604 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:22.885600    4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb2c0928-23fa-462f-baa4-4bd63dab4d62-istio-envoy\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:43:22.975681 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:22.975647    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5"]
Nov 03 22:43:22.980428 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:22.980399    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-cj9f5"]
Nov 03 22:43:23.423934 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:23.423900    4431 patch_prober.go:29] interesting pod/e2e-inf-b-0a7065-openshift-default-56696cf555-sf6vl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Nov 03 22:43:23.424126 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:23.423951    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-sf6vl" podUID="2c772f6e-d9c8-4dbe-aff9-b6b6026032ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Nov 03 22:43:23.506749 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:23.506711    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:43:23.510026 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:23.509998    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-124490-7cbff46899-7c5p6"
Nov 03 22:43:24.089440 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:24.089405    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb2c0928-23fa-462f-baa4-4bd63dab4d62" path="/var/lib/kubelet/pods/bb2c0928-23fa-462f-baa4-4bd63dab4d62/volumes"
Nov 03 22:43:24.427326 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:24.424615    4431 patch_prober.go:29] interesting pod/e2e-inf-b-0a7065-openshift-default-56696cf555-sf6vl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Nov 03 22:43:24.427326 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:24.424675    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-sf6vl" podUID="2c772f6e-d9c8-4dbe-aff9-b6b6026032ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Nov 03 22:43:25.127176 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:25.127142    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:43:25.614259 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:25.614194    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-sf6vl"
Nov 03 22:43:25.614846 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:25.614820    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-sf6vl"
Nov 03 22:43:25.615795 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:25.615776    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-sf6vl"
Nov 03 22:43:26.909912 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:26.909877    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:43:28.496400 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:28.496355    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:43:29.997205 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:29.997165    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:43:31.284764 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:43:31.284724    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:43:31.617760 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:31.617683    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:43:33.211474 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:33.211432    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:43:34.794295 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:34.794256    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:43:36.625286 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:36.625241    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:43:36.746026 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:36.745990    4431 generic.go:356] "Generic (PLEG): container finished" podID="79d7b9fc-6282-4aa1-bae1-e8cbe554eadf" containerID="cf8ef89b306aecedf3c8c429d71f4a272f85645abec9c84d410be06c528f3acd" exitCode=0
Nov 03 22:43:36.746026 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:36.746028    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz" event={"ID":"79d7b9fc-6282-4aa1-bae1-e8cbe554eadf","Type":"ContainerDied","Data":"cf8ef89b306aecedf3c8c429d71f4a272f85645abec9c84d410be06c528f3acd"}
Nov 03 22:43:36.746512 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:36.746494    4431 scope.go:122] "RemoveContainer" containerID="cf8ef89b306aecedf3c8c429d71f4a272f85645abec9c84d410be06c528f3acd"
Nov 03 22:43:36.748513 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:36.748485    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:43:36.748585 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:36.748555    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:43:36.748621 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:36.748589    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:43:37.268623 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:37.268589    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:43:37.784766 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:37.784728    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz" event={"ID":"79d7b9fc-6282-4aa1-bae1-e8cbe554eadf","Type":"ContainerStarted","Data":"5d95bdef12b7500251ab1fe06c11ced7f26d2cf546469eda91dc44b5df3a3bac"}
Nov 03 22:43:38.209181 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:38.209141    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:43:38.326154 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:38.325728    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz"
Nov 03 22:43:38.326154 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:38.325846    4431 patch_prober.go:29] interesting pod/e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Nov 03 22:43:38.326154 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:38.325901    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz" podUID="79d7b9fc-6282-4aa1-bae1-e8cbe554eadf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Nov 03 22:43:39.293500 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:39.293455    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:43:39.325584 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:39.325547    4431 patch_prober.go:29] interesting pod/e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Nov 03 22:43:39.325748 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:39.325645    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz" podUID="79d7b9fc-6282-4aa1-bae1-e8cbe554eadf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Nov 03 22:43:39.920888 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:39.920839    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:43:40.325829 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:40.325722    4431 patch_prober.go:29] interesting pod/e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Nov 03 22:43:40.325829 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:40.325773    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz" podUID="79d7b9fc-6282-4aa1-bae1-e8cbe554eadf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Nov 03 22:43:41.325259 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:41.325195    4431 patch_prober.go:29] interesting pod/e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Nov 03 22:43:41.325447 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:41.325295    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz" podUID="79d7b9fc-6282-4aa1-bae1-e8cbe554eadf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Nov 03 22:43:41.504720 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:41.504684    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:43:42.325999 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:42.325961    4431 patch_prober.go:29] interesting pod/e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Nov 03 22:43:42.326210 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:42.326021    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz" podUID="79d7b9fc-6282-4aa1-bae1-e8cbe554eadf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Nov 03 22:43:42.996398 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:42.996359    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:43:43.330652 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:43.330573    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz"
Nov 03 22:43:43.331046 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:43.331026    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz"
Nov 03 22:43:43.331816 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:43.331795    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz"
Nov 03 22:43:44.458465 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:44.458427    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:43:46.078820 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:46.078788    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:43:46.228958 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:46.228920    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:43:46.434971 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:46.434937    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:43:47.803047 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:47.803005    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:43:47.947370 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:47.947338    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:43:49.383847 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:49.383812    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:43:51.006370 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:51.006334    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:43:52.597082 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:52.597040    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:43:52.943279 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:52.943245    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:43:54.260224 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:54.260175    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:43:55.874067 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:55.874033    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:43:56.030342 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:56.030304    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:43:56.175252 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:56.175198    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:43:57.512566 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:57.512524    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:43:59.434103 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:43:59.434066    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:44:01.207011 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:01.206961    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:44:01.277545 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:44:01.277510    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:44:02.706076 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:02.706012    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:44:04.437466 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:04.437428    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:44:05.324224 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:05.324171    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:44:06.309793 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:06.309758    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:44:07.256330 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:07.256291    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:44:07.832196 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:07.832167    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:44:09.284782 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:09.284747    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:44:09.866413 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:09.866369    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:44:10.004684 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:10.004651    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:44:10.856816 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:10.856782    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:44:12.456049 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:12.455997    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:44:12.982203 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:12.981816    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:44:14.225415 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:14.225383    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:44:15.720387 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:15.720344    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:44:17.501502 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:17.501465    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:44:18.519687 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:18.519653    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-50b9b5/test-model-50b9b5-kserve-6c4d7fbdfc-clplg"]
Nov 03 22:44:18.520690 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:18.520665    4431 cpu_manager.go:414] "containerMap: removing container" podUID="bb2c0928-23fa-462f-baa4-4bd63dab4d62" containerName="istio-proxy"
Nov 03 22:44:18.520690 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:18.520689    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb2c0928-23fa-462f-baa4-4bd63dab4d62" containerName="istio-proxy"
Nov 03 22:44:18.520902 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:18.520887    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb2c0928-23fa-462f-baa4-4bd63dab4d62" containerName="istio-proxy"
Nov 03 22:44:18.531364 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:18.531336    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-50b9b5/test-model-50b9b5-kserve-6c4d7fbdfc-clplg"
Nov 03 22:44:18.551734 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:18.551707    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-50b9b5/test-model-50b9b5-kserve-6c4d7fbdfc-clplg"]
Nov 03 22:44:18.556932 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:18.556905    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-50b9b5\"/\"kube-root-ca.crt\""
Nov 03 22:44:18.557039 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:18.556905    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-50b9b5\"/\"openshift-service-ca.crt\""
Nov 03 22:44:18.557560 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:18.557541    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-50b9b5\"/\"default-dockercfg-xxdst\""
Nov 03 22:44:18.557692 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:18.557671    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-50b9b5\"/\"test-model-50b9b5-kserve-self-signed-certs\""
Nov 03 22:44:18.610714 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:18.610676    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:44:18.662091 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:18.659687    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79bf63c8-226a-410a-933e-c97971c11629-model-cache\") pod \"test-model-50b9b5-kserve-6c4d7fbdfc-clplg\" (UID: \"79bf63c8-226a-410a-933e-c97971c11629\") " pod="ai-tenant-e2e-inf-a-50b9b5/test-model-50b9b5-kserve-6c4d7fbdfc-clplg"
Nov 03 22:44:18.662091 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:18.659905    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79bf63c8-226a-410a-933e-c97971c11629-tmp-dir\") pod \"test-model-50b9b5-kserve-6c4d7fbdfc-clplg\" (UID: \"79bf63c8-226a-410a-933e-c97971c11629\") " pod="ai-tenant-e2e-inf-a-50b9b5/test-model-50b9b5-kserve-6c4d7fbdfc-clplg"
Nov 03 22:44:18.662091 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:18.660052    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79bf63c8-226a-410a-933e-c97971c11629-tls-certs\") pod \"test-model-50b9b5-kserve-6c4d7fbdfc-clplg\" (UID: \"79bf63c8-226a-410a-933e-c97971c11629\") " pod="ai-tenant-e2e-inf-a-50b9b5/test-model-50b9b5-kserve-6c4d7fbdfc-clplg"
Nov 03 22:44:18.662091 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:18.660115    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79bf63c8-226a-410a-933e-c97971c11629-home\") pod \"test-model-50b9b5-kserve-6c4d7fbdfc-clplg\" (UID: \"79bf63c8-226a-410a-933e-c97971c11629\") " pod="ai-tenant-e2e-inf-a-50b9b5/test-model-50b9b5-kserve-6c4d7fbdfc-clplg"
Nov 03 22:44:18.662091 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:18.660188    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq4c8\" (UniqueName: \"kubernetes.io/projected/79bf63c8-226a-410a-933e-c97971c11629-kube-api-access-bq4c8\") pod \"test-model-50b9b5-kserve-6c4d7fbdfc-clplg\" (UID: \"79bf63c8-226a-410a-933e-c97971c11629\") " pod="ai-tenant-e2e-inf-a-50b9b5/test-model-50b9b5-kserve-6c4d7fbdfc-clplg"
Nov 03 22:44:18.662091 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:18.660246    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79bf63c8-226a-410a-933e-c97971c11629-dshm\") pod \"test-model-50b9b5-kserve-6c4d7fbdfc-clplg\" (UID: \"79bf63c8-226a-410a-933e-c97971c11629\") " pod="ai-tenant-e2e-inf-a-50b9b5/test-model-50b9b5-kserve-6c4d7fbdfc-clplg"
Nov 03 22:44:18.764237 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:18.762351    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79bf63c8-226a-410a-933e-c97971c11629-tmp-dir\") pod \"test-model-50b9b5-kserve-6c4d7fbdfc-clplg\" (UID: \"79bf63c8-226a-410a-933e-c97971c11629\") " pod="ai-tenant-e2e-inf-a-50b9b5/test-model-50b9b5-kserve-6c4d7fbdfc-clplg"
Nov 03 22:44:18.764237 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:18.762514    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79bf63c8-226a-410a-933e-c97971c11629-tls-certs\") pod \"test-model-50b9b5-kserve-6c4d7fbdfc-clplg\" (UID: \"79bf63c8-226a-410a-933e-c97971c11629\") " pod="ai-tenant-e2e-inf-a-50b9b5/test-model-50b9b5-kserve-6c4d7fbdfc-clplg"
Nov 03 22:44:18.764237 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:18.762565    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79bf63c8-226a-410a-933e-c97971c11629-home\") pod \"test-model-50b9b5-kserve-6c4d7fbdfc-clplg\" (UID: \"79bf63c8-226a-410a-933e-c97971c11629\") " pod="ai-tenant-e2e-inf-a-50b9b5/test-model-50b9b5-kserve-6c4d7fbdfc-clplg"
Nov 03 22:44:18.764237 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:18.762633    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq4c8\" (UniqueName: \"kubernetes.io/projected/79bf63c8-226a-410a-933e-c97971c11629-kube-api-access-bq4c8\") pod \"test-model-50b9b5-kserve-6c4d7fbdfc-clplg\" (UID: \"79bf63c8-226a-410a-933e-c97971c11629\") " pod="ai-tenant-e2e-inf-a-50b9b5/test-model-50b9b5-kserve-6c4d7fbdfc-clplg"
Nov 03 22:44:18.764237 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:18.762672    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79bf63c8-226a-410a-933e-c97971c11629-dshm\") pod \"test-model-50b9b5-kserve-6c4d7fbdfc-clplg\" (UID: \"79bf63c8-226a-410a-933e-c97971c11629\") " pod="ai-tenant-e2e-inf-a-50b9b5/test-model-50b9b5-kserve-6c4d7fbdfc-clplg"
Nov 03 22:44:18.764237 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:18.762772    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79bf63c8-226a-410a-933e-c97971c11629-model-cache\") pod \"test-model-50b9b5-kserve-6c4d7fbdfc-clplg\" (UID: \"79bf63c8-226a-410a-933e-c97971c11629\") " pod="ai-tenant-e2e-inf-a-50b9b5/test-model-50b9b5-kserve-6c4d7fbdfc-clplg"
Nov 03 22:44:18.764237 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:18.763004    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79bf63c8-226a-410a-933e-c97971c11629-tmp-dir\") pod \"test-model-50b9b5-kserve-6c4d7fbdfc-clplg\" (UID: \"79bf63c8-226a-410a-933e-c97971c11629\") " pod="ai-tenant-e2e-inf-a-50b9b5/test-model-50b9b5-kserve-6c4d7fbdfc-clplg"
Nov 03 22:44:18.764237 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:18.763229    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79bf63c8-226a-410a-933e-c97971c11629-model-cache\") pod \"test-model-50b9b5-kserve-6c4d7fbdfc-clplg\" (UID: \"79bf63c8-226a-410a-933e-c97971c11629\") " pod="ai-tenant-e2e-inf-a-50b9b5/test-model-50b9b5-kserve-6c4d7fbdfc-clplg"
Nov 03 22:44:18.764237 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:18.764204    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79bf63c8-226a-410a-933e-c97971c11629-home\") pod \"test-model-50b9b5-kserve-6c4d7fbdfc-clplg\" (UID: \"79bf63c8-226a-410a-933e-c97971c11629\") " pod="ai-tenant-e2e-inf-a-50b9b5/test-model-50b9b5-kserve-6c4d7fbdfc-clplg"
Nov 03 22:44:18.825310 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:18.824980    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79bf63c8-226a-410a-933e-c97971c11629-tls-certs\") pod \"test-model-50b9b5-kserve-6c4d7fbdfc-clplg\" (UID: \"79bf63c8-226a-410a-933e-c97971c11629\") " pod="ai-tenant-e2e-inf-a-50b9b5/test-model-50b9b5-kserve-6c4d7fbdfc-clplg"
Nov 03 22:44:18.825310 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:18.825208    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79bf63c8-226a-410a-933e-c97971c11629-dshm\") pod \"test-model-50b9b5-kserve-6c4d7fbdfc-clplg\" (UID: \"79bf63c8-226a-410a-933e-c97971c11629\") " pod="ai-tenant-e2e-inf-a-50b9b5/test-model-50b9b5-kserve-6c4d7fbdfc-clplg"
Nov 03 22:44:18.855246 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:18.849932    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq4c8\" (UniqueName: \"kubernetes.io/projected/79bf63c8-226a-410a-933e-c97971c11629-kube-api-access-bq4c8\") pod \"test-model-50b9b5-kserve-6c4d7fbdfc-clplg\" (UID: \"79bf63c8-226a-410a-933e-c97971c11629\") " pod="ai-tenant-e2e-inf-a-50b9b5/test-model-50b9b5-kserve-6c4d7fbdfc-clplg"
Nov 03 22:44:18.855246 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:18.850252    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-50b9b5/test-model-50b9b5-kserve-6c4d7fbdfc-clplg"
Nov 03 22:44:19.021167 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:19.020828    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:44:19.273402 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:19.273369    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-50b9b5/test-model-50b9b5-kserve-6c4d7fbdfc-clplg"]
Nov 03 22:44:19.273643 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:44:19.273618    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod79bf63c8_226a_410a_933e_c97971c11629.slice/crio-298d7f4c549ba000c1953eb99183aecb34b9a2f570f7411185a6124854602316 WatchSource:0}: Error finding container 298d7f4c549ba000c1953eb99183aecb34b9a2f570f7411185a6124854602316: Status 404 returned error can't find the container with id 298d7f4c549ba000c1953eb99183aecb34b9a2f570f7411185a6124854602316
Nov 03 22:44:20.275904 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:20.275869    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-50b9b5/test-model-50b9b5-kserve-6c4d7fbdfc-clplg" event={"ID":"79bf63c8-226a-410a-933e-c97971c11629","Type":"ContainerStarted","Data":"45d4762d98d546502e6b26196aa08fec11a8fa4f658bc83be456a7f897f67aad"}
Nov 03 22:44:20.275904 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:20.275910    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-50b9b5/test-model-50b9b5-kserve-6c4d7fbdfc-clplg" event={"ID":"79bf63c8-226a-410a-933e-c97971c11629","Type":"ContainerStarted","Data":"298d7f4c549ba000c1953eb99183aecb34b9a2f570f7411185a6124854602316"}
Nov 03 22:44:20.318081 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:20.318032    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-50b9b5/test-model-50b9b5-kserve-6c4d7fbdfc-clplg" podStartSLOduration=2.318016948 podStartE2EDuration="2.318016948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:44:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:44:20.31651518 +0000 UTC m=+1884.342371043" watchObservedRunningTime="2026-11-03 22:44:20.318016948 +0000 UTC m=+1884.343872809"
Nov 03 22:44:20.636299 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:20.634593    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:44:20.855830 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:20.855795    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-50b9b5/test-model-50b9b5-kserve-6c4d7fbdfc-clplg"
Nov 03 22:44:22.343032 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:22.338425    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:44:22.652818 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:22.652736    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:44:22.817741 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:22.817706    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:44:23.801238 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:23.794380    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:44:25.368497 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:25.368464    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:44:27.107374 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:27.107332    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:44:28.580202 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:28.580166    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:44:28.856540 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:28.856460    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-50b9b5/test-model-50b9b5-kserve-6c4d7fbdfc-clplg"
Nov 03 22:44:28.869470 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:28.869447    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-50b9b5/test-model-50b9b5-kserve-6c4d7fbdfc-clplg"
Nov 03 22:44:29.358895 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:29.358867    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-50b9b5/test-model-50b9b5-kserve-6c4d7fbdfc-clplg"
Nov 03 22:44:30.141131 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:30.141097    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:44:31.268503 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:44:31.268456    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:44:31.833920 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:31.833881    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:44:33.305782 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:33.305749    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:44:34.768342 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:34.768309    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:44:36.437305 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:36.437273    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:44:38.075794 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:38.075758    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:44:38.216358 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:38.216296    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:44:39.899887 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:39.899856    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:44:41.351975 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:41.351938    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:44:42.168735 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:42.168701    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-fe7ca0/test-model-fe7ca0-kserve-5b69659fbc-jpdrg"]
Nov 03 22:44:42.173745 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:42.173722    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-fe7ca0/test-model-fe7ca0-kserve-5b69659fbc-jpdrg"
Nov 03 22:44:42.203899 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:42.203868    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-fe7ca0\"/\"kube-root-ca.crt\""
Nov 03 22:44:42.204115 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:42.203953    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-fe7ca0\"/\"test-model-fe7ca0-kserve-self-signed-certs\""
Nov 03 22:44:42.204115 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:42.203979    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-fe7ca0\"/\"default-dockercfg-qrmqh\""
Nov 03 22:44:42.204115 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:42.203997    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-fe7ca0\"/\"openshift-service-ca.crt\""
Nov 03 22:44:42.204871 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:42.204840    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f04425d-fb46-454f-84af-dd29b0208c21-model-cache\") pod \"test-model-fe7ca0-kserve-5b69659fbc-jpdrg\" (UID: \"4f04425d-fb46-454f-84af-dd29b0208c21\") " pod="ai-tenant-e2e-inf-b-fe7ca0/test-model-fe7ca0-kserve-5b69659fbc-jpdrg"
Nov 03 22:44:42.205010 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:42.204988    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f04425d-fb46-454f-84af-dd29b0208c21-home\") pod \"test-model-fe7ca0-kserve-5b69659fbc-jpdrg\" (UID: \"4f04425d-fb46-454f-84af-dd29b0208c21\") " pod="ai-tenant-e2e-inf-b-fe7ca0/test-model-fe7ca0-kserve-5b69659fbc-jpdrg"
Nov 03 22:44:42.205134 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:42.205050    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f04425d-fb46-454f-84af-dd29b0208c21-tls-certs\") pod \"test-model-fe7ca0-kserve-5b69659fbc-jpdrg\" (UID: \"4f04425d-fb46-454f-84af-dd29b0208c21\") " pod="ai-tenant-e2e-inf-b-fe7ca0/test-model-fe7ca0-kserve-5b69659fbc-jpdrg"
Nov 03 22:44:42.205199 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:42.205152    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sn6r7\" (UniqueName: \"kubernetes.io/projected/4f04425d-fb46-454f-84af-dd29b0208c21-kube-api-access-sn6r7\") pod \"test-model-fe7ca0-kserve-5b69659fbc-jpdrg\" (UID: \"4f04425d-fb46-454f-84af-dd29b0208c21\") " pod="ai-tenant-e2e-inf-b-fe7ca0/test-model-fe7ca0-kserve-5b69659fbc-jpdrg"
Nov 03 22:44:42.205434 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:42.205414    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f04425d-fb46-454f-84af-dd29b0208c21-tmp-dir\") pod \"test-model-fe7ca0-kserve-5b69659fbc-jpdrg\" (UID: \"4f04425d-fb46-454f-84af-dd29b0208c21\") " pod="ai-tenant-e2e-inf-b-fe7ca0/test-model-fe7ca0-kserve-5b69659fbc-jpdrg"
Nov 03 22:44:42.205531 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:42.205486    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f04425d-fb46-454f-84af-dd29b0208c21-dshm\") pod \"test-model-fe7ca0-kserve-5b69659fbc-jpdrg\" (UID: \"4f04425d-fb46-454f-84af-dd29b0208c21\") " pod="ai-tenant-e2e-inf-b-fe7ca0/test-model-fe7ca0-kserve-5b69659fbc-jpdrg"
Nov 03 22:44:42.209229 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:42.209191    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-fe7ca0/test-model-fe7ca0-kserve-5b69659fbc-jpdrg"]
Nov 03 22:44:42.251897 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:42.251865    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:44:42.306647 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:42.306602    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f04425d-fb46-454f-84af-dd29b0208c21-model-cache\") pod \"test-model-fe7ca0-kserve-5b69659fbc-jpdrg\" (UID: \"4f04425d-fb46-454f-84af-dd29b0208c21\") " pod="ai-tenant-e2e-inf-b-fe7ca0/test-model-fe7ca0-kserve-5b69659fbc-jpdrg"
Nov 03 22:44:42.306847 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:42.306750    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f04425d-fb46-454f-84af-dd29b0208c21-home\") pod \"test-model-fe7ca0-kserve-5b69659fbc-jpdrg\" (UID: \"4f04425d-fb46-454f-84af-dd29b0208c21\") " pod="ai-tenant-e2e-inf-b-fe7ca0/test-model-fe7ca0-kserve-5b69659fbc-jpdrg"
Nov 03 22:44:42.306847 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:42.306806    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f04425d-fb46-454f-84af-dd29b0208c21-tls-certs\") pod \"test-model-fe7ca0-kserve-5b69659fbc-jpdrg\" (UID: \"4f04425d-fb46-454f-84af-dd29b0208c21\") " pod="ai-tenant-e2e-inf-b-fe7ca0/test-model-fe7ca0-kserve-5b69659fbc-jpdrg"
Nov 03 22:44:42.306967 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:42.306912    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sn6r7\" (UniqueName: \"kubernetes.io/projected/4f04425d-fb46-454f-84af-dd29b0208c21-kube-api-access-sn6r7\") pod \"test-model-fe7ca0-kserve-5b69659fbc-jpdrg\" (UID: \"4f04425d-fb46-454f-84af-dd29b0208c21\") " pod="ai-tenant-e2e-inf-b-fe7ca0/test-model-fe7ca0-kserve-5b69659fbc-jpdrg"
Nov 03 22:44:42.307070 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:42.307036    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f04425d-fb46-454f-84af-dd29b0208c21-tmp-dir\") pod \"test-model-fe7ca0-kserve-5b69659fbc-jpdrg\" (UID: \"4f04425d-fb46-454f-84af-dd29b0208c21\") " pod="ai-tenant-e2e-inf-b-fe7ca0/test-model-fe7ca0-kserve-5b69659fbc-jpdrg"
Nov 03 22:44:42.307163 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:42.307089    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f04425d-fb46-454f-84af-dd29b0208c21-model-cache\") pod \"test-model-fe7ca0-kserve-5b69659fbc-jpdrg\" (UID: \"4f04425d-fb46-454f-84af-dd29b0208c21\") " pod="ai-tenant-e2e-inf-b-fe7ca0/test-model-fe7ca0-kserve-5b69659fbc-jpdrg"
Nov 03 22:44:42.307163 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:42.307112    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f04425d-fb46-454f-84af-dd29b0208c21-dshm\") pod \"test-model-fe7ca0-kserve-5b69659fbc-jpdrg\" (UID: \"4f04425d-fb46-454f-84af-dd29b0208c21\") " pod="ai-tenant-e2e-inf-b-fe7ca0/test-model-fe7ca0-kserve-5b69659fbc-jpdrg"
Nov 03 22:44:42.307163 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:42.307124    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f04425d-fb46-454f-84af-dd29b0208c21-home\") pod \"test-model-fe7ca0-kserve-5b69659fbc-jpdrg\" (UID: \"4f04425d-fb46-454f-84af-dd29b0208c21\") " pod="ai-tenant-e2e-inf-b-fe7ca0/test-model-fe7ca0-kserve-5b69659fbc-jpdrg"
Nov 03 22:44:42.307398 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:42.307377    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f04425d-fb46-454f-84af-dd29b0208c21-tmp-dir\") pod \"test-model-fe7ca0-kserve-5b69659fbc-jpdrg\" (UID: \"4f04425d-fb46-454f-84af-dd29b0208c21\") " pod="ai-tenant-e2e-inf-b-fe7ca0/test-model-fe7ca0-kserve-5b69659fbc-jpdrg"
Nov 03 22:44:42.309355 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:42.309328    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f04425d-fb46-454f-84af-dd29b0208c21-dshm\") pod \"test-model-fe7ca0-kserve-5b69659fbc-jpdrg\" (UID: \"4f04425d-fb46-454f-84af-dd29b0208c21\") " pod="ai-tenant-e2e-inf-b-fe7ca0/test-model-fe7ca0-kserve-5b69659fbc-jpdrg"
Nov 03 22:44:42.309925 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:42.309906    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f04425d-fb46-454f-84af-dd29b0208c21-tls-certs\") pod \"test-model-fe7ca0-kserve-5b69659fbc-jpdrg\" (UID: \"4f04425d-fb46-454f-84af-dd29b0208c21\") " pod="ai-tenant-e2e-inf-b-fe7ca0/test-model-fe7ca0-kserve-5b69659fbc-jpdrg"
Nov 03 22:44:42.330180 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:42.330150    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sn6r7\" (UniqueName: \"kubernetes.io/projected/4f04425d-fb46-454f-84af-dd29b0208c21-kube-api-access-sn6r7\") pod \"test-model-fe7ca0-kserve-5b69659fbc-jpdrg\" (UID: \"4f04425d-fb46-454f-84af-dd29b0208c21\") " pod="ai-tenant-e2e-inf-b-fe7ca0/test-model-fe7ca0-kserve-5b69659fbc-jpdrg"
Nov 03 22:44:42.493473 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:42.493380    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-fe7ca0/test-model-fe7ca0-kserve-5b69659fbc-jpdrg"
Nov 03 22:44:42.861601 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:42.861571    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-fe7ca0/test-model-fe7ca0-kserve-5b69659fbc-jpdrg"]
Nov 03 22:44:42.862456 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:44:42.862427    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4f04425d_fb46_454f_84af_dd29b0208c21.slice/crio-b69cdc8840803a05865a71af65b673d727909a1baf08837dad96e7dd418c9ade WatchSource:0}: Error finding container b69cdc8840803a05865a71af65b673d727909a1baf08837dad96e7dd418c9ade: Status 404 returned error can't find the container with id b69cdc8840803a05865a71af65b673d727909a1baf08837dad96e7dd418c9ade
Nov 03 22:44:43.024379 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:43.024342    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:44:43.473398 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:43.473356    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-fe7ca0/test-model-fe7ca0-kserve-5b69659fbc-jpdrg" event={"ID":"4f04425d-fb46-454f-84af-dd29b0208c21","Type":"ContainerStarted","Data":"532e3fd87d90edf7a10ac1656eabbc2280231647a15a4a9a5743cadd305a8dc1"}
Nov 03 22:44:43.473398 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:43.473403    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-fe7ca0/test-model-fe7ca0-kserve-5b69659fbc-jpdrg" event={"ID":"4f04425d-fb46-454f-84af-dd29b0208c21","Type":"ContainerStarted","Data":"b69cdc8840803a05865a71af65b673d727909a1baf08837dad96e7dd418c9ade"}
Nov 03 22:44:43.495088 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:43.495058    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-fe7ca0/test-model-fe7ca0-kserve-5b69659fbc-jpdrg"
Nov 03 22:44:43.517412 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:43.517363    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-fe7ca0/test-model-fe7ca0-kserve-5b69659fbc-jpdrg" podStartSLOduration=1.517348566 podStartE2EDuration="1.517348566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:44:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:44:43.51475624 +0000 UTC m=+1907.540612103" watchObservedRunningTime="2026-11-03 22:44:43.517348566 +0000 UTC m=+1907.543204428"
Nov 03 22:44:44.725240 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:44.725187    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:44:46.442592 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:46.442545    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:44:48.010286 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:48.010254    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:44:48.159612 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:48.159571    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:44:48.321886 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:48.321769    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:44:49.805503 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:49.805464    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:44:51.415735 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:51.415696    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:44:51.561524 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:51.561486    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:44:51.712294 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:51.711715    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:44:52.494339 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:52.494305    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-fe7ca0/test-model-fe7ca0-kserve-5b69659fbc-jpdrg"
Nov 03 22:44:52.512438 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:52.512409    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-fe7ca0/test-model-fe7ca0-kserve-5b69659fbc-jpdrg"
Nov 03 22:44:52.567967 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:52.567936    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-fe7ca0/test-model-fe7ca0-kserve-5b69659fbc-jpdrg"
Nov 03 22:44:53.071606 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:53.071569    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:44:53.271881 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:53.271840    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:44:55.118151 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:55.118108    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:44:57.095482 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:57.095433    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:44:58.849176 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:44:58.849070    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:45:00.348549 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.348513    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a940d5-29895765-bwkkx"]
Nov 03 22:45:00.353763 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.353734    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a940d5-29895765-bwkkx"
Nov 03 22:45:00.354256 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.354209    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0a7065-29895765-2x2gq"]
Nov 03 22:45:00.359796 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.359774    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0a7065-29895765-2x2gq"
Nov 03 22:45:00.379467 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.379288    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\""
Nov 03 22:45:00.401144 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.401104    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gdzq\" (UniqueName: \"kubernetes.io/projected/f511dde2-8f9b-4670-bd73-b7ee34d69706-kube-api-access-2gdzq\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-a940d5-29895765-bwkkx\" (UID: \"f511dde2-8f9b-4670-bd73-b7ee34d69706\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a940d5-29895765-bwkkx"
Nov 03 22:45:00.401465 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.401441    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ef57864-5184-4b21-8fae-201006734abf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-0a7065-29895765-2x2gq\" (UID: \"7ef57864-5184-4b21-8fae-201006734abf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0a7065-29895765-2x2gq"
Nov 03 22:45:00.408073 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.402868    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a940d5-29895765-bwkkx"]
Nov 03 22:45:00.408073 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.407184    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7940ea-29895765-nvrxh"]
Nov 03 22:45:00.408577 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.408526    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f511dde2-8f9b-4670-bd73-b7ee34d69706-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-a940d5-29895765-bwkkx\" (UID: \"f511dde2-8f9b-4670-bd73-b7ee34d69706\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a940d5-29895765-bwkkx"
Nov 03 22:45:00.408843 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.408817    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k72qh\" (UniqueName: \"kubernetes.io/projected/7ef57864-5184-4b21-8fae-201006734abf-kube-api-access-k72qh\") pod \"maas-api-key-cleanup-e2e-inf-b-0a7065-29895765-2x2gq\" (UID: \"7ef57864-5184-4b21-8fae-201006734abf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0a7065-29895765-2x2gq"
Nov 03 22:45:00.412004 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.411983    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7940ea-29895765-nvrxh"
Nov 03 22:45:00.429378 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.419166    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895765-66lpj"]
Nov 03 22:45:00.429378 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.425767    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895765-66lpj"
Nov 03 22:45:00.434866 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.434833    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895765-7fxjw"]
Nov 03 22:45:00.448332 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.448309    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0a7065-29895765-2x2gq"]
Nov 03 22:45:00.448662 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.448643    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895765-7fxjw"
Nov 03 22:45:00.456195 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.456053    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7940ea-29895765-nvrxh"]
Nov 03 22:45:00.461927 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.461900    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-434ce3-29895765-247jr"]
Nov 03 22:45:00.477903 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.477240    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895765-7fxjw"]
Nov 03 22:45:00.477903 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.477358    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-434ce3-29895765-247jr"
Nov 03 22:45:00.483990 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.483954    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2d619f-29895765-29qhp"]
Nov 03 22:45:00.507319 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.497316    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2d619f-29895765-29qhp"
Nov 03 22:45:00.507319 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.504319    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895765-66lpj"]
Nov 03 22:45:00.510617 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.510586    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d201ce0-fde0-494b-988a-b19d31dbbcda-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895765-7fxjw\" (UID: \"0d201ce0-fde0-494b-988a-b19d31dbbcda\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895765-7fxjw"
Nov 03 22:45:00.512955 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.510643    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbvpc\" (UniqueName: \"kubernetes.io/projected/96676e8f-4e63-42e0-88b3-5588c9c2fed3-kube-api-access-kbvpc\") pod \"maas-api-key-cleanup-29895765-66lpj\" (UID: \"96676e8f-4e63-42e0-88b3-5588c9c2fed3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895765-66lpj"
Nov 03 22:45:00.512955 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.510777    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f511dde2-8f9b-4670-bd73-b7ee34d69706-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-a940d5-29895765-bwkkx\" (UID: \"f511dde2-8f9b-4670-bd73-b7ee34d69706\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a940d5-29895765-bwkkx"
Nov 03 22:45:00.512955 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.510827    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/600936df-e54c-429b-ad28-e30a6a76af8f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-434ce3-29895765-247jr\" (UID: \"600936df-e54c-429b-ad28-e30a6a76af8f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-434ce3-29895765-247jr"
Nov 03 22:45:00.512955 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.510917    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k72qh\" (UniqueName: \"kubernetes.io/projected/7ef57864-5184-4b21-8fae-201006734abf-kube-api-access-k72qh\") pod \"maas-api-key-cleanup-e2e-inf-b-0a7065-29895765-2x2gq\" (UID: \"7ef57864-5184-4b21-8fae-201006734abf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0a7065-29895765-2x2gq"
Nov 03 22:45:00.512955 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.511021    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gdzq\" (UniqueName: \"kubernetes.io/projected/f511dde2-8f9b-4670-bd73-b7ee34d69706-kube-api-access-2gdzq\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-a940d5-29895765-bwkkx\" (UID: \"f511dde2-8f9b-4670-bd73-b7ee34d69706\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a940d5-29895765-bwkkx"
Nov 03 22:45:00.512955 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.511059    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csp9s\" (UniqueName: \"kubernetes.io/projected/0d201ce0-fde0-494b-988a-b19d31dbbcda-kube-api-access-csp9s\") pod \"maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895765-7fxjw\" (UID: \"0d201ce0-fde0-494b-988a-b19d31dbbcda\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895765-7fxjw"
Nov 03 22:45:00.512955 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.511162    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b6f92d6-3fbb-48fe-854f-6b5b899f56d1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-7940ea-29895765-nvrxh\" (UID: \"4b6f92d6-3fbb-48fe-854f-6b5b899f56d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7940ea-29895765-nvrxh"
Nov 03 22:45:00.512955 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.512369    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f511dde2-8f9b-4670-bd73-b7ee34d69706-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-a940d5-29895765-bwkkx\" (UID: \"f511dde2-8f9b-4670-bd73-b7ee34d69706\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a940d5-29895765-bwkkx"
Nov 03 22:45:00.512955 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.511207    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8pqq\" (UniqueName: \"kubernetes.io/projected/4b6f92d6-3fbb-48fe-854f-6b5b899f56d1-kube-api-access-h8pqq\") pod \"maas-api-key-cleanup-e2e-inf-a-7940ea-29895765-nvrxh\" (UID: \"4b6f92d6-3fbb-48fe-854f-6b5b899f56d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7940ea-29895765-nvrxh"
Nov 03 22:45:00.512955 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.512747    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmzhd\" (UniqueName: \"kubernetes.io/projected/600936df-e54c-429b-ad28-e30a6a76af8f-kube-api-access-pmzhd\") pod \"maas-api-key-cleanup-e2e-worker-w5-434ce3-29895765-247jr\" (UID: \"600936df-e54c-429b-ad28-e30a6a76af8f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-434ce3-29895765-247jr"
Nov 03 22:45:00.512955 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.512816    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96676e8f-4e63-42e0-88b3-5588c9c2fed3-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895765-66lpj\" (UID: \"96676e8f-4e63-42e0-88b3-5588c9c2fed3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895765-66lpj"
Nov 03 22:45:00.512955 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.512863    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ef57864-5184-4b21-8fae-201006734abf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-0a7065-29895765-2x2gq\" (UID: \"7ef57864-5184-4b21-8fae-201006734abf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0a7065-29895765-2x2gq"
Nov 03 22:45:00.514502 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.514481    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ef57864-5184-4b21-8fae-201006734abf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-0a7065-29895765-2x2gq\" (UID: \"7ef57864-5184-4b21-8fae-201006734abf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0a7065-29895765-2x2gq"
Nov 03 22:45:00.536290 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.534519    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-963936-29895765-52lrg"]
Nov 03 22:45:00.540513 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.540492    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-963936-29895765-52lrg"
Nov 03 22:45:00.543620 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.543588    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-434ce3-29895765-247jr"]
Nov 03 22:45:00.555202 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.555138    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gdzq\" (UniqueName: \"kubernetes.io/projected/f511dde2-8f9b-4670-bd73-b7ee34d69706-kube-api-access-2gdzq\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-a940d5-29895765-bwkkx\" (UID: \"f511dde2-8f9b-4670-bd73-b7ee34d69706\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a940d5-29895765-bwkkx"
Nov 03 22:45:00.555750 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.555697    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k72qh\" (UniqueName: \"kubernetes.io/projected/7ef57864-5184-4b21-8fae-201006734abf-kube-api-access-k72qh\") pod \"maas-api-key-cleanup-e2e-inf-b-0a7065-29895765-2x2gq\" (UID: \"7ef57864-5184-4b21-8fae-201006734abf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0a7065-29895765-2x2gq"
Nov 03 22:45:00.593294 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.570237    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2d6076-29895765-5njwc"]
Nov 03 22:45:00.619357 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.615707    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2d6076-29895765-5njwc"
Nov 03 22:45:00.625277 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.620862    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csp9s\" (UniqueName: \"kubernetes.io/projected/0d201ce0-fde0-494b-988a-b19d31dbbcda-kube-api-access-csp9s\") pod \"maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895765-7fxjw\" (UID: \"0d201ce0-fde0-494b-988a-b19d31dbbcda\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895765-7fxjw"
Nov 03 22:45:00.625277 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.620973    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfml7\" (UniqueName: \"kubernetes.io/projected/cc558639-1e15-4789-aaae-d6535a4e62e8-kube-api-access-dfml7\") pod \"maas-api-key-cleanup-e2e-worker-w6-2d619f-29895765-29qhp\" (UID: \"cc558639-1e15-4789-aaae-d6535a4e62e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2d619f-29895765-29qhp"
Nov 03 22:45:00.625277 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.621024    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b6f92d6-3fbb-48fe-854f-6b5b899f56d1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-7940ea-29895765-nvrxh\" (UID: \"4b6f92d6-3fbb-48fe-854f-6b5b899f56d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7940ea-29895765-nvrxh"
Nov 03 22:45:00.625277 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.621062    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc558639-1e15-4789-aaae-d6535a4e62e8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-2d619f-29895765-29qhp\" (UID: \"cc558639-1e15-4789-aaae-d6535a4e62e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2d619f-29895765-29qhp"
Nov 03 22:45:00.625277 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.621092    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8pqq\" (UniqueName: \"kubernetes.io/projected/4b6f92d6-3fbb-48fe-854f-6b5b899f56d1-kube-api-access-h8pqq\") pod \"maas-api-key-cleanup-e2e-inf-a-7940ea-29895765-nvrxh\" (UID: \"4b6f92d6-3fbb-48fe-854f-6b5b899f56d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7940ea-29895765-nvrxh"
Nov 03 22:45:00.625277 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.621131    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmzhd\" (UniqueName: \"kubernetes.io/projected/600936df-e54c-429b-ad28-e30a6a76af8f-kube-api-access-pmzhd\") pod \"maas-api-key-cleanup-e2e-worker-w5-434ce3-29895765-247jr\" (UID: \"600936df-e54c-429b-ad28-e30a6a76af8f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-434ce3-29895765-247jr"
Nov 03 22:45:00.625277 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.621204    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96676e8f-4e63-42e0-88b3-5588c9c2fed3-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895765-66lpj\" (UID: \"96676e8f-4e63-42e0-88b3-5588c9c2fed3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895765-66lpj"
Nov 03 22:45:00.625277 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.621350    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d201ce0-fde0-494b-988a-b19d31dbbcda-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895765-7fxjw\" (UID: \"0d201ce0-fde0-494b-988a-b19d31dbbcda\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895765-7fxjw"
Nov 03 22:45:00.625277 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.621399    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbvpc\" (UniqueName: \"kubernetes.io/projected/96676e8f-4e63-42e0-88b3-5588c9c2fed3-kube-api-access-kbvpc\") pod \"maas-api-key-cleanup-29895765-66lpj\" (UID: \"96676e8f-4e63-42e0-88b3-5588c9c2fed3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895765-66lpj"
Nov 03 22:45:00.625277 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.621544    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/600936df-e54c-429b-ad28-e30a6a76af8f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-434ce3-29895765-247jr\" (UID: \"600936df-e54c-429b-ad28-e30a6a76af8f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-434ce3-29895765-247jr"
Nov 03 22:45:00.625277 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.621623    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv46j\" (UniqueName: \"kubernetes.io/projected/57ada6e6-75cf-47a4-87d6-8ad44c0e442e-kube-api-access-hv46j\") pod \"maas-api-key-cleanup-e2e-inf-b-963936-29895765-52lrg\" (UID: \"57ada6e6-75cf-47a4-87d6-8ad44c0e442e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-963936-29895765-52lrg"
Nov 03 22:45:00.625277 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.621682    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57ada6e6-75cf-47a4-87d6-8ad44c0e442e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-963936-29895765-52lrg\" (UID: \"57ada6e6-75cf-47a4-87d6-8ad44c0e442e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-963936-29895765-52lrg"
Nov 03 22:45:00.625277 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.623193    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b6f92d6-3fbb-48fe-854f-6b5b899f56d1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-7940ea-29895765-nvrxh\" (UID: \"4b6f92d6-3fbb-48fe-854f-6b5b899f56d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7940ea-29895765-nvrxh"
Nov 03 22:45:00.625277 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.623432    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96676e8f-4e63-42e0-88b3-5588c9c2fed3-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895765-66lpj\" (UID: \"96676e8f-4e63-42e0-88b3-5588c9c2fed3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895765-66lpj"
Nov 03 22:45:00.625277 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.624104    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d201ce0-fde0-494b-988a-b19d31dbbcda-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895765-7fxjw\" (UID: \"0d201ce0-fde0-494b-988a-b19d31dbbcda\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895765-7fxjw"
Nov 03 22:45:00.625277 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.625027    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/600936df-e54c-429b-ad28-e30a6a76af8f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-434ce3-29895765-247jr\" (UID: \"600936df-e54c-429b-ad28-e30a6a76af8f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-434ce3-29895765-247jr"
Nov 03 22:45:00.638873 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.635530    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b1ec3a-29895765-s8js9"]
Nov 03 22:45:00.638873 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.637153    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:45:00.650698 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.650391    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmzhd\" (UniqueName: \"kubernetes.io/projected/600936df-e54c-429b-ad28-e30a6a76af8f-kube-api-access-pmzhd\") pod \"maas-api-key-cleanup-e2e-worker-w5-434ce3-29895765-247jr\" (UID: \"600936df-e54c-429b-ad28-e30a6a76af8f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-434ce3-29895765-247jr"
Nov 03 22:45:00.662237 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.652878    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b1ec3a-29895765-s8js9"
Nov 03 22:45:00.662237 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.654700    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csp9s\" (UniqueName: \"kubernetes.io/projected/0d201ce0-fde0-494b-988a-b19d31dbbcda-kube-api-access-csp9s\") pod \"maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895765-7fxjw\" (UID: \"0d201ce0-fde0-494b-988a-b19d31dbbcda\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895765-7fxjw"
Nov 03 22:45:00.668515 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.668479    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8pqq\" (UniqueName: \"kubernetes.io/projected/4b6f92d6-3fbb-48fe-854f-6b5b899f56d1-kube-api-access-h8pqq\") pod \"maas-api-key-cleanup-e2e-inf-a-7940ea-29895765-nvrxh\" (UID: \"4b6f92d6-3fbb-48fe-854f-6b5b899f56d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7940ea-29895765-nvrxh"
Nov 03 22:45:00.668683 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.668559    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-963936-29895765-52lrg"]
Nov 03 22:45:00.677499 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.677474    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbvpc\" (UniqueName: \"kubernetes.io/projected/96676e8f-4e63-42e0-88b3-5588c9c2fed3-kube-api-access-kbvpc\") pod \"maas-api-key-cleanup-29895765-66lpj\" (UID: \"96676e8f-4e63-42e0-88b3-5588c9c2fed3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895765-66lpj"
Nov 03 22:45:00.682104 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.682066    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2d619f-29895765-29qhp"]
Nov 03 22:45:00.687582 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.687550    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2d6076-29895765-5njwc"]
Nov 03 22:45:00.692518 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.692493    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b1ec3a-29895765-s8js9"]
Nov 03 22:45:00.696473 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.696451    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a940d5-29895765-bwkkx"
Nov 03 22:45:00.705974 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.705347    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0a7065-29895765-2x2gq"
Nov 03 22:45:00.723698 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.723667    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv46j\" (UniqueName: \"kubernetes.io/projected/57ada6e6-75cf-47a4-87d6-8ad44c0e442e-kube-api-access-hv46j\") pod \"maas-api-key-cleanup-e2e-inf-b-963936-29895765-52lrg\" (UID: \"57ada6e6-75cf-47a4-87d6-8ad44c0e442e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-963936-29895765-52lrg"
Nov 03 22:45:00.723830 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.723739    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7kcp\" (UniqueName: \"kubernetes.io/projected/8b5f2047-8624-42fb-9881-6d81c25867ee-kube-api-access-r7kcp\") pod \"maas-api-key-cleanup-e2e-inf-b-2d6076-29895765-5njwc\" (UID: \"8b5f2047-8624-42fb-9881-6d81c25867ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2d6076-29895765-5njwc"
Nov 03 22:45:00.723830 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.723773    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57ada6e6-75cf-47a4-87d6-8ad44c0e442e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-963936-29895765-52lrg\" (UID: \"57ada6e6-75cf-47a4-87d6-8ad44c0e442e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-963936-29895765-52lrg"
Nov 03 22:45:00.723830 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.723808    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9a74dcb-40e3-415a-a090-3619177efa8a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-b1ec3a-29895765-s8js9\" (UID: \"f9a74dcb-40e3-415a-a090-3619177efa8a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b1ec3a-29895765-s8js9"
Nov 03 22:45:00.724045 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.723939    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qhrz\" (UniqueName: \"kubernetes.io/projected/f9a74dcb-40e3-415a-a090-3619177efa8a-kube-api-access-7qhrz\") pod \"maas-api-key-cleanup-e2e-inf-a-b1ec3a-29895765-s8js9\" (UID: \"f9a74dcb-40e3-415a-a090-3619177efa8a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b1ec3a-29895765-s8js9"
Nov 03 22:45:00.725634 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.724412    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfml7\" (UniqueName: \"kubernetes.io/projected/cc558639-1e15-4789-aaae-d6535a4e62e8-kube-api-access-dfml7\") pod \"maas-api-key-cleanup-e2e-worker-w6-2d619f-29895765-29qhp\" (UID: \"cc558639-1e15-4789-aaae-d6535a4e62e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2d619f-29895765-29qhp"
Nov 03 22:45:00.725634 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.724466    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b5f2047-8624-42fb-9881-6d81c25867ee-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-2d6076-29895765-5njwc\" (UID: \"8b5f2047-8624-42fb-9881-6d81c25867ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2d6076-29895765-5njwc"
Nov 03 22:45:00.725634 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.724517    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc558639-1e15-4789-aaae-d6535a4e62e8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-2d619f-29895765-29qhp\" (UID: \"cc558639-1e15-4789-aaae-d6535a4e62e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2d619f-29895765-29qhp"
Nov 03 22:45:00.725634 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.725347    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc558639-1e15-4789-aaae-d6535a4e62e8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-2d619f-29895765-29qhp\" (UID: \"cc558639-1e15-4789-aaae-d6535a4e62e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2d619f-29895765-29qhp"
Nov 03 22:45:00.725634 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.725435    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57ada6e6-75cf-47a4-87d6-8ad44c0e442e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-963936-29895765-52lrg\" (UID: \"57ada6e6-75cf-47a4-87d6-8ad44c0e442e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-963936-29895765-52lrg"
Nov 03 22:45:00.738692 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.738665    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7940ea-29895765-nvrxh"
Nov 03 22:45:00.753964 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.753922    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895765-66lpj"
Nov 03 22:45:00.774135 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.774105    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfml7\" (UniqueName: \"kubernetes.io/projected/cc558639-1e15-4789-aaae-d6535a4e62e8-kube-api-access-dfml7\") pod \"maas-api-key-cleanup-e2e-worker-w6-2d619f-29895765-29qhp\" (UID: \"cc558639-1e15-4789-aaae-d6535a4e62e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2d619f-29895765-29qhp"
Nov 03 22:45:00.780896 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.780499    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895765-7fxjw"
Nov 03 22:45:00.794577 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.793678    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv46j\" (UniqueName: \"kubernetes.io/projected/57ada6e6-75cf-47a4-87d6-8ad44c0e442e-kube-api-access-hv46j\") pod \"maas-api-key-cleanup-e2e-inf-b-963936-29895765-52lrg\" (UID: \"57ada6e6-75cf-47a4-87d6-8ad44c0e442e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-963936-29895765-52lrg"
Nov 03 22:45:00.843236 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.840282    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-434ce3-29895765-247jr"
Nov 03 22:45:00.843236 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.840907    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7kcp\" (UniqueName: \"kubernetes.io/projected/8b5f2047-8624-42fb-9881-6d81c25867ee-kube-api-access-r7kcp\") pod \"maas-api-key-cleanup-e2e-inf-b-2d6076-29895765-5njwc\" (UID: \"8b5f2047-8624-42fb-9881-6d81c25867ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2d6076-29895765-5njwc"
Nov 03 22:45:00.843236 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.840952    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9a74dcb-40e3-415a-a090-3619177efa8a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-b1ec3a-29895765-s8js9\" (UID: \"f9a74dcb-40e3-415a-a090-3619177efa8a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b1ec3a-29895765-s8js9"
Nov 03 22:45:00.843236 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.841059    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qhrz\" (UniqueName: \"kubernetes.io/projected/f9a74dcb-40e3-415a-a090-3619177efa8a-kube-api-access-7qhrz\") pod \"maas-api-key-cleanup-e2e-inf-a-b1ec3a-29895765-s8js9\" (UID: \"f9a74dcb-40e3-415a-a090-3619177efa8a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b1ec3a-29895765-s8js9"
Nov 03 22:45:00.843236 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.841145    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b5f2047-8624-42fb-9881-6d81c25867ee-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-2d6076-29895765-5njwc\" (UID: \"8b5f2047-8624-42fb-9881-6d81c25867ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2d6076-29895765-5njwc"
Nov 03 22:45:00.843236 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.842045    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b5f2047-8624-42fb-9881-6d81c25867ee-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-2d6076-29895765-5njwc\" (UID: \"8b5f2047-8624-42fb-9881-6d81c25867ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2d6076-29895765-5njwc"
Nov 03 22:45:00.843236 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.842632    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9a74dcb-40e3-415a-a090-3619177efa8a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-b1ec3a-29895765-s8js9\" (UID: \"f9a74dcb-40e3-415a-a090-3619177efa8a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b1ec3a-29895765-s8js9"
Nov 03 22:45:00.860171 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.857355    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2d619f-29895765-29qhp"
Nov 03 22:45:00.894094 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.894010    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7kcp\" (UniqueName: \"kubernetes.io/projected/8b5f2047-8624-42fb-9881-6d81c25867ee-kube-api-access-r7kcp\") pod \"maas-api-key-cleanup-e2e-inf-b-2d6076-29895765-5njwc\" (UID: \"8b5f2047-8624-42fb-9881-6d81c25867ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2d6076-29895765-5njwc"
Nov 03 22:45:00.901435 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.901376    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qhrz\" (UniqueName: \"kubernetes.io/projected/f9a74dcb-40e3-415a-a090-3619177efa8a-kube-api-access-7qhrz\") pod \"maas-api-key-cleanup-e2e-inf-a-b1ec3a-29895765-s8js9\" (UID: \"f9a74dcb-40e3-415a-a090-3619177efa8a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b1ec3a-29895765-s8js9"
Nov 03 22:45:00.978024 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:00.977956    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-963936-29895765-52lrg"
Nov 03 22:45:01.028946 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:01.028913    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2d6076-29895765-5njwc"
Nov 03 22:45:01.044554 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:01.044515    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b1ec3a-29895765-s8js9"
Nov 03 22:45:01.048697 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:45:01.048664    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ef57864_5184_4b21_8fae_201006734abf.slice/crio-e16beff988df659af008188c5218b5a0b833739f87a7f5c6fd02df6b965b2282 WatchSource:0}: Error finding container e16beff988df659af008188c5218b5a0b833739f87a7f5c6fd02df6b965b2282: Status 404 returned error can't find the container with id e16beff988df659af008188c5218b5a0b833739f87a7f5c6fd02df6b965b2282
Nov 03 22:45:01.049845 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:01.049488    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0a7065-29895765-2x2gq"]
Nov 03 22:45:01.243931 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:01.243198    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a940d5-29895765-bwkkx"]
Nov 03 22:45:01.251650 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:45:01.251618    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf511dde2_8f9b_4670_bd73_b7ee34d69706.slice/crio-05a4eb588a17ccf92c4ac026c884a146ac26aaab8ef1089ec0d004a3764e2789 WatchSource:0}: Error finding container 05a4eb588a17ccf92c4ac026c884a146ac26aaab8ef1089ec0d004a3764e2789: Status 404 returned error can't find the container with id 05a4eb588a17ccf92c4ac026c884a146ac26aaab8ef1089ec0d004a3764e2789
Nov 03 22:45:01.284375 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:45:01.284337    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:45:01.324699 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:01.324674    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895765-66lpj"]
Nov 03 22:45:01.327943 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:45:01.327914    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96676e8f_4e63_42e0_88b3_5588c9c2fed3.slice/crio-61dd653b4d555bf27b98538225f6b23f3bc086c2c5ccf2b9e0d91cbd07bc785f WatchSource:0}: Error finding container 61dd653b4d555bf27b98538225f6b23f3bc086c2c5ccf2b9e0d91cbd07bc785f: Status 404 returned error can't find the container with id 61dd653b4d555bf27b98538225f6b23f3bc086c2c5ccf2b9e0d91cbd07bc785f
Nov 03 22:45:01.332498 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:01.332471    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7940ea-29895765-nvrxh"]
Nov 03 22:45:01.371347 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:01.371264    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895765-7fxjw"]
Nov 03 22:45:01.376267 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:45:01.376199    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d201ce0_fde0_494b_988a_b19d31dbbcda.slice/crio-580100edab54278fe7728359395c74714dcff381ad6ea8da768f7a229f5edfad WatchSource:0}: Error finding container 580100edab54278fe7728359395c74714dcff381ad6ea8da768f7a229f5edfad: Status 404 returned error can't find the container with id 580100edab54278fe7728359395c74714dcff381ad6ea8da768f7a229f5edfad
Nov 03 22:45:01.393945 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:45:01.393917    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b5f2047_8624_42fb_9881_6d81c25867ee.slice/crio-4de9da6ef477cf12e4453f052b9db674689f8439a2e56f81ae12a9f7be3f92e3 WatchSource:0}: Error finding container 4de9da6ef477cf12e4453f052b9db674689f8439a2e56f81ae12a9f7be3f92e3: Status 404 returned error can't find the container with id 4de9da6ef477cf12e4453f052b9db674689f8439a2e56f81ae12a9f7be3f92e3
Nov 03 22:45:01.394069 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:01.394015    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2d6076-29895765-5njwc"]
Nov 03 22:45:01.416753 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:01.416726    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b1ec3a-29895765-s8js9"]
Nov 03 22:45:01.420082 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:45:01.420056    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9a74dcb_40e3_415a_a090_3619177efa8a.slice/crio-93286abb12224f62887f6817bdc542829f06c82e86614dfe2b5e10cb9e3d8f7b WatchSource:0}: Error finding container 93286abb12224f62887f6817bdc542829f06c82e86614dfe2b5e10cb9e3d8f7b: Status 404 returned error can't find the container with id 93286abb12224f62887f6817bdc542829f06c82e86614dfe2b5e10cb9e3d8f7b
Nov 03 22:45:01.421091 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:45:01.421067    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc558639_1e15_4789_aaae_d6535a4e62e8.slice/crio-1b0d1abc4afb42a23c9f9c561d8437e1605708d1ef928a193bb74be296207294 WatchSource:0}: Error finding container 1b0d1abc4afb42a23c9f9c561d8437e1605708d1ef928a193bb74be296207294: Status 404 returned error can't find the container with id 1b0d1abc4afb42a23c9f9c561d8437e1605708d1ef928a193bb74be296207294
Nov 03 22:45:01.422255 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:01.422228    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2d619f-29895765-29qhp"]
Nov 03 22:45:01.460280 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:01.460248    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-434ce3-29895765-247jr"]
Nov 03 22:45:01.460681 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:45:01.460652    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod600936df_e54c_429b_ad28_e30a6a76af8f.slice/crio-8c730eb8aaa519202d3923a3312c69f154bb3e40a1f94e6aff898ae8fbf4028e WatchSource:0}: Error finding container 8c730eb8aaa519202d3923a3312c69f154bb3e40a1f94e6aff898ae8fbf4028e: Status 404 returned error can't find the container with id 8c730eb8aaa519202d3923a3312c69f154bb3e40a1f94e6aff898ae8fbf4028e
Nov 03 22:45:01.554066 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:01.554037    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-963936-29895765-52lrg"]
Nov 03 22:45:01.555343 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:45:01.555316    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57ada6e6_75cf_47a4_87d6_8ad44c0e442e.slice/crio-d2035bb4c1a686df1b9dd0ea70e7b75c8116d4525c8f94773248fb08a78e4887 WatchSource:0}: Error finding container d2035bb4c1a686df1b9dd0ea70e7b75c8116d4525c8f94773248fb08a78e4887: Status 404 returned error can't find the container with id d2035bb4c1a686df1b9dd0ea70e7b75c8116d4525c8f94773248fb08a78e4887
Nov 03 22:45:01.981562 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:01.981524    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895765-66lpj" event={"ID":"96676e8f-4e63-42e0-88b3-5588c9c2fed3","Type":"ContainerStarted","Data":"61dd653b4d555bf27b98538225f6b23f3bc086c2c5ccf2b9e0d91cbd07bc785f"}
Nov 03 22:45:01.983476 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:01.983443    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-434ce3-29895765-247jr" event={"ID":"600936df-e54c-429b-ad28-e30a6a76af8f","Type":"ContainerStarted","Data":"8c730eb8aaa519202d3923a3312c69f154bb3e40a1f94e6aff898ae8fbf4028e"}
Nov 03 22:45:01.986719 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:01.986685    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0a7065-29895765-2x2gq" event={"ID":"7ef57864-5184-4b21-8fae-201006734abf","Type":"ContainerStarted","Data":"e16beff988df659af008188c5218b5a0b833739f87a7f5c6fd02df6b965b2282"}
Nov 03 22:45:01.990089 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:01.990045    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2d6076-29895765-5njwc" event={"ID":"8b5f2047-8624-42fb-9881-6d81c25867ee","Type":"ContainerStarted","Data":"4de9da6ef477cf12e4453f052b9db674689f8439a2e56f81ae12a9f7be3f92e3"}
Nov 03 22:45:01.991905 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:01.991873    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7940ea-29895765-nvrxh" event={"ID":"4b6f92d6-3fbb-48fe-854f-6b5b899f56d1","Type":"ContainerStarted","Data":"e55d69ec0e5bf96f3a6cd81c16b344e19075f4c4950941e1fa9d7e414911c8f4"}
Nov 03 22:45:01.995756 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:01.995624    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b1ec3a-29895765-s8js9" event={"ID":"f9a74dcb-40e3-415a-a090-3619177efa8a","Type":"ContainerStarted","Data":"93286abb12224f62887f6817bdc542829f06c82e86614dfe2b5e10cb9e3d8f7b"}
Nov 03 22:45:02.001140 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:02.001080    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2d619f-29895765-29qhp" event={"ID":"cc558639-1e15-4789-aaae-d6535a4e62e8","Type":"ContainerStarted","Data":"1b0d1abc4afb42a23c9f9c561d8437e1605708d1ef928a193bb74be296207294"}
Nov 03 22:45:02.005248 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:02.005104    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895765-7fxjw" event={"ID":"0d201ce0-fde0-494b-988a-b19d31dbbcda","Type":"ContainerStarted","Data":"580100edab54278fe7728359395c74714dcff381ad6ea8da768f7a229f5edfad"}
Nov 03 22:45:02.008413 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:02.008382    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a940d5-29895765-bwkkx" event={"ID":"f511dde2-8f9b-4670-bd73-b7ee34d69706","Type":"ContainerStarted","Data":"05a4eb588a17ccf92c4ac026c884a146ac26aaab8ef1089ec0d004a3764e2789"}
Nov 03 22:45:02.011346 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:02.011319    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-963936-29895765-52lrg" event={"ID":"57ada6e6-75cf-47a4-87d6-8ad44c0e442e","Type":"ContainerStarted","Data":"d2035bb4c1a686df1b9dd0ea70e7b75c8116d4525c8f94773248fb08a78e4887"}
Nov 03 22:45:02.597617 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:02.597580    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:45:04.317305 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:04.317258    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:45:04.596270 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:04.595896    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:45:04.760243 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:04.760155    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:45:05.901698 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:05.901640    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:45:07.564491 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:07.564444    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:45:08.264692 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:08.264632    4431 generic.go:356] "Generic (PLEG): container finished" podID="2cbef074-f335-41d5-a69c-b84a958e87ee" containerID="12bd9f676d34dd68d6e917b86ac592cd27ca4690bd71c1ecf29d513985017952" exitCode=0
Nov 03 22:45:08.264692 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:08.264674    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-963936-openshift-default-68b65f857c-phdvv" event={"ID":"2cbef074-f335-41d5-a69c-b84a958e87ee","Type":"ContainerDied","Data":"12bd9f676d34dd68d6e917b86ac592cd27ca4690bd71c1ecf29d513985017952"}
Nov 03 22:45:08.265311 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:08.265261    4431 scope.go:122] "RemoveContainer" containerID="12bd9f676d34dd68d6e917b86ac592cd27ca4690bd71c1ecf29d513985017952"
Nov 03 22:45:08.278085 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:08.277941    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:45:08.278085 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:08.278074    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:45:08.278331 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:08.278135    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:45:09.135356 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:09.135324    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:45:09.275434 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:09.275406    4431 generic.go:356] "Generic (PLEG): container finished" podID="f511dde2-8f9b-4670-bd73-b7ee34d69706" containerID="c150b03c9565fd622106b4cc58fad85b2b39d32d910a8711d44d391aebcc7e30" exitCode=60
Nov 03 22:45:09.275585 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:09.275500    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a940d5-29895765-bwkkx" event={"ID":"f511dde2-8f9b-4670-bd73-b7ee34d69706","Type":"ContainerDied","Data":"c150b03c9565fd622106b4cc58fad85b2b39d32d910a8711d44d391aebcc7e30"}
Nov 03 22:45:09.275801 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:09.275786    4431 scope.go:122] "RemoveContainer" containerID="c150b03c9565fd622106b4cc58fad85b2b39d32d910a8711d44d391aebcc7e30"
Nov 03 22:45:09.277722 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:09.277662    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0a7065-29895765-2x2gq" event={"ID":"7ef57864-5184-4b21-8fae-201006734abf","Type":"ContainerStarted","Data":"46defdf8cedcb9dbdafc32825cbe62ade2b1f483b939b578b310fb43243cf143"}
Nov 03 22:45:09.280732 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:09.280696    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-963936-openshift-default-68b65f857c-phdvv" event={"ID":"2cbef074-f335-41d5-a69c-b84a958e87ee","Type":"ContainerStarted","Data":"ca8359caacf7375447f51fd89147358bd7a1ecd1353d57c6ef1171690d006cf7"}
Nov 03 22:45:09.397785 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:09.397728    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0a7065-29895765-2x2gq" podStartSLOduration=1.8323692280000001 podStartE2EDuration="9.39771108s" totalImagesPullingTime="7.565341852s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:09.395485996 +0000 UTC m=+1933.421341858" watchObservedRunningTime="2026-11-03 22:45:09.39771108 +0000 UTC m=+1933.423566942"
Nov 03 22:45:10.023375 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:10.023290    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-963936-openshift-default-68b65f857c-phdvv"
Nov 03 22:45:10.024385 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:10.024361    4431 patch_prober.go:29] interesting pod/e2e-inf-b-963936-openshift-default-68b65f857c-phdvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Nov 03 22:45:10.024460 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:10.024407    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-963936-openshift-default-68b65f857c-phdvv" podUID="2cbef074-f335-41d5-a69c-b84a958e87ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Nov 03 22:45:10.292904 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:10.292869    4431 generic.go:356] "Generic (PLEG): container finished" podID="f9a74dcb-40e3-415a-a090-3619177efa8a" containerID="24c1ad0659a134722ffd10446b7a1ed2255cf90a35ffe9c886bbfa1add8141a4" exitCode=60
Nov 03 22:45:10.293292 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:10.293035    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b1ec3a-29895765-s8js9" event={"ID":"f9a74dcb-40e3-415a-a090-3619177efa8a","Type":"ContainerDied","Data":"24c1ad0659a134722ffd10446b7a1ed2255cf90a35ffe9c886bbfa1add8141a4"}
Nov 03 22:45:10.293606 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:10.293586    4431 scope.go:122] "RemoveContainer" containerID="24c1ad0659a134722ffd10446b7a1ed2255cf90a35ffe9c886bbfa1add8141a4"
Nov 03 22:45:10.297280 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:10.297247    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2d619f-29895765-29qhp" event={"ID":"cc558639-1e15-4789-aaae-d6535a4e62e8","Type":"ContainerStarted","Data":"ba79bcd1914594aef8dcd071720cbe7784326d16b7c2631c596439e766c910d5"}
Nov 03 22:45:10.303852 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:10.303825    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895765-7fxjw" event={"ID":"0d201ce0-fde0-494b-988a-b19d31dbbcda","Type":"ContainerStarted","Data":"d161dc00e51b7eb8af14b65236b568bb6e200069c2164bfd2b5fea5b43663d23"}
Nov 03 22:45:10.306874 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:10.306816    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a940d5-29895765-bwkkx" event={"ID":"f511dde2-8f9b-4670-bd73-b7ee34d69706","Type":"ContainerStarted","Data":"c8b02f4357cadcb5e5967c8d343a98233fc79b58278275af5ff4b518cee8e783"}
Nov 03 22:45:10.309034 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:10.308927    4431 generic.go:356] "Generic (PLEG): container finished" podID="57ada6e6-75cf-47a4-87d6-8ad44c0e442e" containerID="5d385ff2e35f3a900e65424e6b0d3464d3bdc2c8735d05a7b6d6b34e8e9e20af" exitCode=60
Nov 03 22:45:10.309034 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:10.309013    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-963936-29895765-52lrg" event={"ID":"57ada6e6-75cf-47a4-87d6-8ad44c0e442e","Type":"ContainerDied","Data":"5d385ff2e35f3a900e65424e6b0d3464d3bdc2c8735d05a7b6d6b34e8e9e20af"}
Nov 03 22:45:10.309449 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:10.309186    4431 scope.go:122] "RemoveContainer" containerID="5d385ff2e35f3a900e65424e6b0d3464d3bdc2c8735d05a7b6d6b34e8e9e20af"
Nov 03 22:45:10.311807 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:10.311226    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895765-66lpj" event={"ID":"96676e8f-4e63-42e0-88b3-5588c9c2fed3","Type":"ContainerStarted","Data":"2b40443bdc90831d9347a3f7a975659e76668ed1cbccc7732877e42589cb93b8"}
Nov 03 22:45:10.314198 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:10.313723    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-434ce3-29895765-247jr" event={"ID":"600936df-e54c-429b-ad28-e30a6a76af8f","Type":"ContainerStarted","Data":"f0f87023c0189ecb03281dd0840ea56fd5b74518b1d22d8b83b16860e236e6e6"}
Nov 03 22:45:10.316467 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:10.315960    4431 generic.go:356] "Generic (PLEG): container finished" podID="7ef57864-5184-4b21-8fae-201006734abf" containerID="46defdf8cedcb9dbdafc32825cbe62ade2b1f483b939b578b310fb43243cf143" exitCode=60
Nov 03 22:45:10.316467 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:10.316080    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0a7065-29895765-2x2gq" event={"ID":"7ef57864-5184-4b21-8fae-201006734abf","Type":"ContainerDied","Data":"46defdf8cedcb9dbdafc32825cbe62ade2b1f483b939b578b310fb43243cf143"}
Nov 03 22:45:10.316467 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:10.316187    4431 scope.go:122] "RemoveContainer" containerID="46defdf8cedcb9dbdafc32825cbe62ade2b1f483b939b578b310fb43243cf143"
Nov 03 22:45:10.319828 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:10.319805    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2d6076-29895765-5njwc" event={"ID":"8b5f2047-8624-42fb-9881-6d81c25867ee","Type":"ContainerStarted","Data":"8eddeeab08afb13062d6400f094dd47b76959c4ff96db25bfa414c9d538834ea"}
Nov 03 22:45:10.323501 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:10.323481    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7940ea-29895765-nvrxh" event={"ID":"4b6f92d6-3fbb-48fe-854f-6b5b899f56d1","Type":"ContainerStarted","Data":"e9685998338b8c473f41c9029768e13a22eab38cac59429d24b8adfd82ad39e2"}
Nov 03 22:45:10.452237 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:10.438937    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a940d5-29895765-bwkkx" podStartSLOduration=3.6708415519999997 podStartE2EDuration="10.438918536s" totalImagesPullingTime="6.768076984s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:10.438332595 +0000 UTC m=+1934.464188458" watchObservedRunningTime="2026-11-03 22:45:10.438918536 +0000 UTC m=+1934.464774400"
Nov 03 22:45:10.485510 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:10.485450    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-434ce3-29895765-247jr" podStartSLOduration=2.8186012700000003 podStartE2EDuration="10.485435948s" totalImagesPullingTime="7.666834678s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:10.484814311 +0000 UTC m=+1934.510670189" watchObservedRunningTime="2026-11-03 22:45:10.485435948 +0000 UTC m=+1934.511291810"
Nov 03 22:45:10.524188 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:10.524134    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2d619f-29895765-29qhp" podStartSLOduration=2.8278883759999998 podStartE2EDuration="10.524119024s" totalImagesPullingTime="7.696230648s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:10.52290483 +0000 UTC m=+1934.548760693" watchObservedRunningTime="2026-11-03 22:45:10.524119024 +0000 UTC m=+1934.549974894"
Nov 03 22:45:10.612274 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:10.612148    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895765-66lpj" podStartSLOduration=2.9126139540000002 podStartE2EDuration="10.612130972s" totalImagesPullingTime="7.699517018s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:10.611087822 +0000 UTC m=+1934.636943689" watchObservedRunningTime="2026-11-03 22:45:10.612130972 +0000 UTC m=+1934.637986835"
Nov 03 22:45:10.667252 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:10.658512    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7940ea-29895765-nvrxh" podStartSLOduration=2.9105758760000002 podStartE2EDuration="10.658491594s" totalImagesPullingTime="7.747915718s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:10.655676814 +0000 UTC m=+1934.681532677" watchObservedRunningTime="2026-11-03 22:45:10.658491594 +0000 UTC m=+1934.684347458"
Nov 03 22:45:10.698445 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:10.698392    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895765-7fxjw" podStartSLOduration=2.7161466880000003 podStartE2EDuration="10.698376057s" totalImagesPullingTime="7.982229369s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:10.695600887 +0000 UTC m=+1934.721456749" watchObservedRunningTime="2026-11-03 22:45:10.698376057 +0000 UTC m=+1934.724231919"
Nov 03 22:45:10.726196 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:10.726131    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:45:10.818236 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:10.816486    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2d6076-29895765-5njwc" podStartSLOduration=3.081066046 podStartE2EDuration="10.816464526s" totalImagesPullingTime="7.73539848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:10.773722184 +0000 UTC m=+1934.799578060" watchObservedRunningTime="2026-11-03 22:45:10.816464526 +0000 UTC m=+1934.842320391"
Nov 03 22:45:11.023983 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:11.023954    4431 patch_prober.go:29] interesting pod/e2e-inf-b-963936-openshift-default-68b65f857c-phdvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Nov 03 22:45:11.024163 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:11.024005    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-963936-openshift-default-68b65f857c-phdvv" podUID="2cbef074-f335-41d5-a69c-b84a958e87ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Nov 03 22:45:11.333297 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:11.333260    4431 generic.go:356] "Generic (PLEG): container finished" podID="96676e8f-4e63-42e0-88b3-5588c9c2fed3" containerID="2b40443bdc90831d9347a3f7a975659e76668ed1cbccc7732877e42589cb93b8" exitCode=60
Nov 03 22:45:11.333741 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:11.333346    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895765-66lpj" event={"ID":"96676e8f-4e63-42e0-88b3-5588c9c2fed3","Type":"ContainerDied","Data":"2b40443bdc90831d9347a3f7a975659e76668ed1cbccc7732877e42589cb93b8"}
Nov 03 22:45:11.333741 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:11.333621    4431 scope.go:122] "RemoveContainer" containerID="2b40443bdc90831d9347a3f7a975659e76668ed1cbccc7732877e42589cb93b8"
Nov 03 22:45:11.335162 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:11.335134    4431 generic.go:356] "Generic (PLEG): container finished" podID="600936df-e54c-429b-ad28-e30a6a76af8f" containerID="f0f87023c0189ecb03281dd0840ea56fd5b74518b1d22d8b83b16860e236e6e6" exitCode=60
Nov 03 22:45:11.335313 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:11.335206    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-434ce3-29895765-247jr" event={"ID":"600936df-e54c-429b-ad28-e30a6a76af8f","Type":"ContainerDied","Data":"f0f87023c0189ecb03281dd0840ea56fd5b74518b1d22d8b83b16860e236e6e6"}
Nov 03 22:45:11.335518 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:11.335501    4431 scope.go:122] "RemoveContainer" containerID="f0f87023c0189ecb03281dd0840ea56fd5b74518b1d22d8b83b16860e236e6e6"
Nov 03 22:45:11.337097 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:11.337074    4431 generic.go:356] "Generic (PLEG): container finished" podID="8b5f2047-8624-42fb-9881-6d81c25867ee" containerID="8eddeeab08afb13062d6400f094dd47b76959c4ff96db25bfa414c9d538834ea" exitCode=60
Nov 03 22:45:11.337210 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:11.337155    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2d6076-29895765-5njwc" event={"ID":"8b5f2047-8624-42fb-9881-6d81c25867ee","Type":"ContainerDied","Data":"8eddeeab08afb13062d6400f094dd47b76959c4ff96db25bfa414c9d538834ea"}
Nov 03 22:45:11.337458 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:11.337443    4431 scope.go:122] "RemoveContainer" containerID="8eddeeab08afb13062d6400f094dd47b76959c4ff96db25bfa414c9d538834ea"
Nov 03 22:45:11.376182 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:11.376149    4431 generic.go:356] "Generic (PLEG): container finished" podID="4b6f92d6-3fbb-48fe-854f-6b5b899f56d1" containerID="e9685998338b8c473f41c9029768e13a22eab38cac59429d24b8adfd82ad39e2" exitCode=60
Nov 03 22:45:11.376404 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:11.376360    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7940ea-29895765-nvrxh" event={"ID":"4b6f92d6-3fbb-48fe-854f-6b5b899f56d1","Type":"ContainerDied","Data":"e9685998338b8c473f41c9029768e13a22eab38cac59429d24b8adfd82ad39e2"}
Nov 03 22:45:11.376605 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:11.376587    4431 scope.go:122] "RemoveContainer" containerID="e9685998338b8c473f41c9029768e13a22eab38cac59429d24b8adfd82ad39e2"
Nov 03 22:45:11.383507 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:11.383170    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b1ec3a-29895765-s8js9" event={"ID":"f9a74dcb-40e3-415a-a090-3619177efa8a","Type":"ContainerStarted","Data":"edb71f93f84f2df5e6cada54258b653e8fdffbe6da81d4d5150552dee5caf97e"}
Nov 03 22:45:11.385862 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:11.385835    4431 generic.go:356] "Generic (PLEG): container finished" podID="cc558639-1e15-4789-aaae-d6535a4e62e8" containerID="ba79bcd1914594aef8dcd071720cbe7784326d16b7c2631c596439e766c910d5" exitCode=60
Nov 03 22:45:11.386044 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:11.385919    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2d619f-29895765-29qhp" event={"ID":"cc558639-1e15-4789-aaae-d6535a4e62e8","Type":"ContainerDied","Data":"ba79bcd1914594aef8dcd071720cbe7784326d16b7c2631c596439e766c910d5"}
Nov 03 22:45:11.386185 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:11.386168    4431 scope.go:122] "RemoveContainer" containerID="ba79bcd1914594aef8dcd071720cbe7784326d16b7c2631c596439e766c910d5"
Nov 03 22:45:11.396349 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:11.396324    4431 generic.go:356] "Generic (PLEG): container finished" podID="0d201ce0-fde0-494b-988a-b19d31dbbcda" containerID="d161dc00e51b7eb8af14b65236b568bb6e200069c2164bfd2b5fea5b43663d23" exitCode=60
Nov 03 22:45:11.396473 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:11.396399    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895765-7fxjw" event={"ID":"0d201ce0-fde0-494b-988a-b19d31dbbcda","Type":"ContainerDied","Data":"d161dc00e51b7eb8af14b65236b568bb6e200069c2164bfd2b5fea5b43663d23"}
Nov 03 22:45:11.397715 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:11.396790    4431 scope.go:122] "RemoveContainer" containerID="d161dc00e51b7eb8af14b65236b568bb6e200069c2164bfd2b5fea5b43663d23"
Nov 03 22:45:11.399118 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:11.399008    4431 generic.go:356] "Generic (PLEG): container finished" podID="f511dde2-8f9b-4670-bd73-b7ee34d69706" containerID="c8b02f4357cadcb5e5967c8d343a98233fc79b58278275af5ff4b518cee8e783" exitCode=60
Nov 03 22:45:11.399118 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:11.399046    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a940d5-29895765-bwkkx" event={"ID":"f511dde2-8f9b-4670-bd73-b7ee34d69706","Type":"ContainerDied","Data":"c8b02f4357cadcb5e5967c8d343a98233fc79b58278275af5ff4b518cee8e783"}
Nov 03 22:45:11.399118 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:11.399109    4431 scope.go:122] "RemoveContainer" containerID="c150b03c9565fd622106b4cc58fad85b2b39d32d910a8711d44d391aebcc7e30"
Nov 03 22:45:11.399601 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:11.399544    4431 scope.go:122] "RemoveContainer" containerID="c8b02f4357cadcb5e5967c8d343a98233fc79b58278275af5ff4b518cee8e783"
Nov 03 22:45:11.400071 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:45:11.400046    4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-a940d5-29895765-bwkkx_odh-ai-gateway-infra(f511dde2-8f9b-4670-bd73-b7ee34d69706)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a940d5-29895765-bwkkx" podUID="f511dde2-8f9b-4670-bd73-b7ee34d69706"
Nov 03 22:45:11.671954 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:11.662001    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b1ec3a-29895765-s8js9" podStartSLOduration=4.347749837 podStartE2EDuration="11.66198127s" totalImagesPullingTime="7.314231433s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:11.655491809 +0000 UTC m=+1935.681347673" watchObservedRunningTime="2026-11-03 22:45:11.66198127 +0000 UTC m=+1935.687837133"
Nov 03 22:45:12.023144 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:12.023111    4431 patch_prober.go:29] interesting pod/e2e-inf-b-963936-openshift-default-68b65f857c-phdvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Nov 03 22:45:12.023297 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:12.023156    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-963936-openshift-default-68b65f857c-phdvv" podUID="2cbef074-f335-41d5-a69c-b84a958e87ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Nov 03 22:45:12.410027 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:12.409995    4431 generic.go:356] "Generic (PLEG): container finished" podID="57ada6e6-75cf-47a4-87d6-8ad44c0e442e" containerID="1ad647742f262cc608eee961cf59f4b4d578e800917ca2f3361076563dde9278" exitCode=60
Nov 03 22:45:12.410514 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:12.410083    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-963936-29895765-52lrg" event={"ID":"57ada6e6-75cf-47a4-87d6-8ad44c0e442e","Type":"ContainerDied","Data":"1ad647742f262cc608eee961cf59f4b4d578e800917ca2f3361076563dde9278"}
Nov 03 22:45:12.410514 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:12.410159    4431 scope.go:122] "RemoveContainer" containerID="5d385ff2e35f3a900e65424e6b0d3464d3bdc2c8735d05a7b6d6b34e8e9e20af"
Nov 03 22:45:12.410623 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:12.410607    4431 scope.go:122] "RemoveContainer" containerID="1ad647742f262cc608eee961cf59f4b4d578e800917ca2f3361076563dde9278"
Nov 03 22:45:12.411022 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:45:12.410997    4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-963936-29895765-52lrg_odh-ai-gateway-infra(57ada6e6-75cf-47a4-87d6-8ad44c0e442e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-963936-29895765-52lrg" podUID="57ada6e6-75cf-47a4-87d6-8ad44c0e442e"
Nov 03 22:45:12.412362 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:12.412332    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895765-66lpj" event={"ID":"96676e8f-4e63-42e0-88b3-5588c9c2fed3","Type":"ContainerStarted","Data":"16b7943d79d69ceb0d567fabf7c1c142eca062a2b409a56befc563e56ca95da6"}
Nov 03 22:45:12.414303 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:12.414279    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-434ce3-29895765-247jr" event={"ID":"600936df-e54c-429b-ad28-e30a6a76af8f","Type":"ContainerStarted","Data":"0b69cb61f3990651913af3b11e451c28c8ff680388cacc95d91f054de062654a"}
Nov 03 22:45:12.416593 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:12.416555    4431 generic.go:356] "Generic (PLEG): container finished" podID="7ef57864-5184-4b21-8fae-201006734abf" containerID="11e55cddd05f228273f6ae3c3b12ec29a22c37d2e53844e8574f26ff44a615c0" exitCode=60
Nov 03 22:45:12.416703 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:12.416634    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0a7065-29895765-2x2gq" event={"ID":"7ef57864-5184-4b21-8fae-201006734abf","Type":"ContainerDied","Data":"11e55cddd05f228273f6ae3c3b12ec29a22c37d2e53844e8574f26ff44a615c0"}
Nov 03 22:45:12.416847 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:12.416827    4431 scope.go:122] "RemoveContainer" containerID="11e55cddd05f228273f6ae3c3b12ec29a22c37d2e53844e8574f26ff44a615c0"
Nov 03 22:45:12.417128 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:45:12.417107    4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-0a7065-29895765-2x2gq_odh-ai-gateway-infra(7ef57864-5184-4b21-8fae-201006734abf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0a7065-29895765-2x2gq" podUID="7ef57864-5184-4b21-8fae-201006734abf"
Nov 03 22:45:12.418721 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:12.418697    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2d6076-29895765-5njwc" event={"ID":"8b5f2047-8624-42fb-9881-6d81c25867ee","Type":"ContainerStarted","Data":"e84fbcea49eaa20eadc6ca58936e17ea2c3a68ea2bcf33ca507ee386fb7f884e"}
Nov 03 22:45:12.420746 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:12.420722    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7940ea-29895765-nvrxh" event={"ID":"4b6f92d6-3fbb-48fe-854f-6b5b899f56d1","Type":"ContainerStarted","Data":"3f58bda8f9ae34e5a4da9bde0f08c7f9607bab226a8f6082d2fd9d2bae71493a"}
Nov 03 22:45:12.422568 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:12.422541    4431 generic.go:356] "Generic (PLEG): container finished" podID="f9a74dcb-40e3-415a-a090-3619177efa8a" containerID="edb71f93f84f2df5e6cada54258b653e8fdffbe6da81d4d5150552dee5caf97e" exitCode=60
Nov 03 22:45:12.422642 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:12.422586    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b1ec3a-29895765-s8js9" event={"ID":"f9a74dcb-40e3-415a-a090-3619177efa8a","Type":"ContainerDied","Data":"edb71f93f84f2df5e6cada54258b653e8fdffbe6da81d4d5150552dee5caf97e"}
Nov 03 22:45:12.422843 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:12.422827    4431 scope.go:122] "RemoveContainer" containerID="edb71f93f84f2df5e6cada54258b653e8fdffbe6da81d4d5150552dee5caf97e"
Nov 03 22:45:12.423131 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:45:12.423101    4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-b1ec3a-29895765-s8js9_odh-ai-gateway-infra(f9a74dcb-40e3-415a-a090-3619177efa8a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b1ec3a-29895765-s8js9" podUID="f9a74dcb-40e3-415a-a090-3619177efa8a"
Nov 03 22:45:12.424596 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:12.424573    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2d619f-29895765-29qhp" event={"ID":"cc558639-1e15-4789-aaae-d6535a4e62e8","Type":"ContainerStarted","Data":"0d99ffe5900f31df4e6bbcb2cfda9dd24e59c881f88f48e9aa21e170b236f166"}
Nov 03 22:45:12.426434 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:12.426406    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895765-7fxjw" event={"ID":"0d201ce0-fde0-494b-988a-b19d31dbbcda","Type":"ContainerStarted","Data":"b5a1e8129c605466aa500f815dc3cf73f5f8157377d6e238c700740b06d5a51f"}
Nov 03 22:45:12.426868 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:12.426849    4431 scope.go:122] "RemoveContainer" containerID="46defdf8cedcb9dbdafc32825cbe62ade2b1f483b939b578b310fb43243cf143"
Nov 03 22:45:12.428504 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:12.428483    4431 scope.go:122] "RemoveContainer" containerID="c8b02f4357cadcb5e5967c8d343a98233fc79b58278275af5ff4b518cee8e783"
Nov 03 22:45:12.428715 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:45:12.428697    4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-a940d5-29895765-bwkkx_odh-ai-gateway-infra(f511dde2-8f9b-4670-bd73-b7ee34d69706)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a940d5-29895765-bwkkx" podUID="f511dde2-8f9b-4670-bd73-b7ee34d69706"
Nov 03 22:45:12.474255 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:12.473623    4431 scope.go:122] "RemoveContainer" containerID="24c1ad0659a134722ffd10446b7a1ed2255cf90a35ffe9c886bbfa1add8141a4"
Nov 03 22:45:12.513705 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:12.513267    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:45:13.023749 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:13.023718    4431 patch_prober.go:29] interesting pod/e2e-inf-b-963936-openshift-default-68b65f857c-phdvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Nov 03 22:45:13.023883 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:13.023777    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-963936-openshift-default-68b65f857c-phdvv" podUID="2cbef074-f335-41d5-a69c-b84a958e87ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Nov 03 22:45:13.463253 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:13.450151    4431 generic.go:356] "Generic (PLEG): container finished" podID="96676e8f-4e63-42e0-88b3-5588c9c2fed3" containerID="16b7943d79d69ceb0d567fabf7c1c142eca062a2b409a56befc563e56ca95da6" exitCode=60
Nov 03 22:45:13.463253 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:13.450282    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895765-66lpj" event={"ID":"96676e8f-4e63-42e0-88b3-5588c9c2fed3","Type":"ContainerDied","Data":"16b7943d79d69ceb0d567fabf7c1c142eca062a2b409a56befc563e56ca95da6"}
Nov 03 22:45:13.463253 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:13.450378    4431 scope.go:122] "RemoveContainer" containerID="2b40443bdc90831d9347a3f7a975659e76668ed1cbccc7732877e42589cb93b8"
Nov 03 22:45:13.463253 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:13.450925    4431 scope.go:122] "RemoveContainer" containerID="16b7943d79d69ceb0d567fabf7c1c142eca062a2b409a56befc563e56ca95da6"
Nov 03 22:45:13.463253 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:45:13.451397    4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895765-66lpj_odh-ai-gateway-infra(96676e8f-4e63-42e0-88b3-5588c9c2fed3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895765-66lpj" podUID="96676e8f-4e63-42e0-88b3-5588c9c2fed3"
Nov 03 22:45:13.468556 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:13.468521    4431 generic.go:356] "Generic (PLEG): container finished" podID="600936df-e54c-429b-ad28-e30a6a76af8f" containerID="0b69cb61f3990651913af3b11e451c28c8ff680388cacc95d91f054de062654a" exitCode=60
Nov 03 22:45:13.468702 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:13.468623    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-434ce3-29895765-247jr" event={"ID":"600936df-e54c-429b-ad28-e30a6a76af8f","Type":"ContainerDied","Data":"0b69cb61f3990651913af3b11e451c28c8ff680388cacc95d91f054de062654a"}
Nov 03 22:45:13.469029 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:13.468992    4431 scope.go:122] "RemoveContainer" containerID="0b69cb61f3990651913af3b11e451c28c8ff680388cacc95d91f054de062654a"
Nov 03 22:45:13.469632 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:45:13.469606    4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-434ce3-29895765-247jr_odh-ai-gateway-infra(600936df-e54c-429b-ad28-e30a6a76af8f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-434ce3-29895765-247jr" podUID="600936df-e54c-429b-ad28-e30a6a76af8f"
Nov 03 22:45:13.472429 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:13.472403    4431 scope.go:122] "RemoveContainer" containerID="11e55cddd05f228273f6ae3c3b12ec29a22c37d2e53844e8574f26ff44a615c0"
Nov 03 22:45:13.472734 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:45:13.472711    4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-0a7065-29895765-2x2gq_odh-ai-gateway-infra(7ef57864-5184-4b21-8fae-201006734abf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0a7065-29895765-2x2gq" podUID="7ef57864-5184-4b21-8fae-201006734abf"
Nov 03 22:45:13.474161 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:13.474139    4431 generic.go:356] "Generic (PLEG): container finished" podID="8b5f2047-8624-42fb-9881-6d81c25867ee" containerID="e84fbcea49eaa20eadc6ca58936e17ea2c3a68ea2bcf33ca507ee386fb7f884e" exitCode=60
Nov 03 22:45:13.474273 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:13.474230    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2d6076-29895765-5njwc" event={"ID":"8b5f2047-8624-42fb-9881-6d81c25867ee","Type":"ContainerDied","Data":"e84fbcea49eaa20eadc6ca58936e17ea2c3a68ea2bcf33ca507ee386fb7f884e"}
Nov 03 22:45:13.474569 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:13.474544    4431 scope.go:122] "RemoveContainer" containerID="e84fbcea49eaa20eadc6ca58936e17ea2c3a68ea2bcf33ca507ee386fb7f884e"
Nov 03 22:45:13.474911 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:45:13.474891    4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-2d6076-29895765-5njwc_odh-ai-gateway-infra(8b5f2047-8624-42fb-9881-6d81c25867ee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2d6076-29895765-5njwc" podUID="8b5f2047-8624-42fb-9881-6d81c25867ee"
Nov 03 22:45:13.476407 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:13.476386    4431 generic.go:356] "Generic (PLEG): container finished" podID="4b6f92d6-3fbb-48fe-854f-6b5b899f56d1" containerID="3f58bda8f9ae34e5a4da9bde0f08c7f9607bab226a8f6082d2fd9d2bae71493a" exitCode=60
Nov 03 22:45:13.476536 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:13.476514    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7940ea-29895765-nvrxh" event={"ID":"4b6f92d6-3fbb-48fe-854f-6b5b899f56d1","Type":"ContainerDied","Data":"3f58bda8f9ae34e5a4da9bde0f08c7f9607bab226a8f6082d2fd9d2bae71493a"}
Nov 03 22:45:13.476721 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:13.476700    4431 scope.go:122] "RemoveContainer" containerID="3f58bda8f9ae34e5a4da9bde0f08c7f9607bab226a8f6082d2fd9d2bae71493a"
Nov 03 22:45:13.477048 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:45:13.477023    4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-7940ea-29895765-nvrxh_odh-ai-gateway-infra(4b6f92d6-3fbb-48fe-854f-6b5b899f56d1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7940ea-29895765-nvrxh" podUID="4b6f92d6-3fbb-48fe-854f-6b5b899f56d1"
Nov 03 22:45:13.478771 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:13.478744    4431 scope.go:122] "RemoveContainer" containerID="edb71f93f84f2df5e6cada54258b653e8fdffbe6da81d4d5150552dee5caf97e"
Nov 03 22:45:13.479074 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:45:13.479051    4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-b1ec3a-29895765-s8js9_odh-ai-gateway-infra(f9a74dcb-40e3-415a-a090-3619177efa8a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b1ec3a-29895765-s8js9" podUID="f9a74dcb-40e3-415a-a090-3619177efa8a"
Nov 03 22:45:13.480711 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:13.480694    4431 scope.go:122] "RemoveContainer" containerID="f0f87023c0189ecb03281dd0840ea56fd5b74518b1d22d8b83b16860e236e6e6"
Nov 03 22:45:13.480971 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:13.480948    4431 generic.go:356] "Generic (PLEG): container finished" podID="cc558639-1e15-4789-aaae-d6535a4e62e8" containerID="0d99ffe5900f31df4e6bbcb2cfda9dd24e59c881f88f48e9aa21e170b236f166" exitCode=60
Nov 03 22:45:13.481141 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:13.480991    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2d619f-29895765-29qhp" event={"ID":"cc558639-1e15-4789-aaae-d6535a4e62e8","Type":"ContainerDied","Data":"0d99ffe5900f31df4e6bbcb2cfda9dd24e59c881f88f48e9aa21e170b236f166"}
Nov 03 22:45:13.481298 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:13.481279    4431 scope.go:122] "RemoveContainer" containerID="0d99ffe5900f31df4e6bbcb2cfda9dd24e59c881f88f48e9aa21e170b236f166"
Nov 03 22:45:13.481594 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:45:13.481536    4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-2d619f-29895765-29qhp_odh-ai-gateway-infra(cc558639-1e15-4789-aaae-d6535a4e62e8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2d619f-29895765-29qhp" podUID="cc558639-1e15-4789-aaae-d6535a4e62e8"
Nov 03 22:45:13.483227 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:13.483198    4431 generic.go:356] "Generic (PLEG): container finished" podID="0d201ce0-fde0-494b-988a-b19d31dbbcda" containerID="b5a1e8129c605466aa500f815dc3cf73f5f8157377d6e238c700740b06d5a51f" exitCode=60
Nov 03 22:45:13.483328 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:13.483305    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895765-7fxjw" event={"ID":"0d201ce0-fde0-494b-988a-b19d31dbbcda","Type":"ContainerDied","Data":"b5a1e8129c605466aa500f815dc3cf73f5f8157377d6e238c700740b06d5a51f"}
Nov 03 22:45:13.483501 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:13.483486    4431 scope.go:122] "RemoveContainer" containerID="b5a1e8129c605466aa500f815dc3cf73f5f8157377d6e238c700740b06d5a51f"
Nov 03 22:45:13.483755 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:45:13.483732    4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895765-7fxjw_odh-ai-gateway-infra(0d201ce0-fde0-494b-988a-b19d31dbbcda)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895765-7fxjw" podUID="0d201ce0-fde0-494b-988a-b19d31dbbcda"
Nov 03 22:45:13.485344 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:13.485327    4431 scope.go:122] "RemoveContainer" containerID="1ad647742f262cc608eee961cf59f4b4d578e800917ca2f3361076563dde9278"
Nov 03 22:45:13.485573 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:45:13.485550    4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-963936-29895765-52lrg_odh-ai-gateway-infra(57ada6e6-75cf-47a4-87d6-8ad44c0e442e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-963936-29895765-52lrg" podUID="57ada6e6-75cf-47a4-87d6-8ad44c0e442e"
Nov 03 22:45:13.496340 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:13.496320    4431 scope.go:122] "RemoveContainer" containerID="8eddeeab08afb13062d6400f094dd47b76959c4ff96db25bfa414c9d538834ea"
Nov 03 22:45:13.511199 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:13.511180    4431 scope.go:122] "RemoveContainer" containerID="e9685998338b8c473f41c9029768e13a22eab38cac59429d24b8adfd82ad39e2"
Nov 03 22:45:13.525093 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:13.525079    4431 scope.go:122] "RemoveContainer" containerID="ba79bcd1914594aef8dcd071720cbe7784326d16b7c2631c596439e766c910d5"
Nov 03 22:45:13.565502 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:13.565476    4431 scope.go:122] "RemoveContainer" containerID="d161dc00e51b7eb8af14b65236b568bb6e200069c2164bfd2b5fea5b43663d23"
Nov 03 22:45:14.023427 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:14.023389    4431 patch_prober.go:29] interesting pod/e2e-inf-b-963936-openshift-default-68b65f857c-phdvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Nov 03 22:45:14.023594 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:14.023453    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-963936-openshift-default-68b65f857c-phdvv" podUID="2cbef074-f335-41d5-a69c-b84a958e87ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Nov 03 22:45:14.311313 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:14.311231    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-963936-openshift-default-68b65f857c-phdvv"
Nov 03 22:45:14.365903 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:14.355636    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:45:14.494322 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:14.494294    4431 scope.go:122] "RemoveContainer" containerID="e84fbcea49eaa20eadc6ca58936e17ea2c3a68ea2bcf33ca507ee386fb7f884e"
Nov 03 22:45:14.494769 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:45:14.494709    4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-2d6076-29895765-5njwc_odh-ai-gateway-infra(8b5f2047-8624-42fb-9881-6d81c25867ee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2d6076-29895765-5njwc" podUID="8b5f2047-8624-42fb-9881-6d81c25867ee"
Nov 03 22:45:14.496064 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:14.496046    4431 scope.go:122] "RemoveContainer" containerID="3f58bda8f9ae34e5a4da9bde0f08c7f9607bab226a8f6082d2fd9d2bae71493a"
Nov 03 22:45:14.496431 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:45:14.496408    4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-7940ea-29895765-nvrxh_odh-ai-gateway-infra(4b6f92d6-3fbb-48fe-854f-6b5b899f56d1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7940ea-29895765-nvrxh" podUID="4b6f92d6-3fbb-48fe-854f-6b5b899f56d1"
Nov 03 22:45:14.497680 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:14.497656    4431 scope.go:122] "RemoveContainer" containerID="0d99ffe5900f31df4e6bbcb2cfda9dd24e59c881f88f48e9aa21e170b236f166"
Nov 03 22:45:14.497941 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:45:14.497923    4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-2d619f-29895765-29qhp_odh-ai-gateway-infra(cc558639-1e15-4789-aaae-d6535a4e62e8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2d619f-29895765-29qhp" podUID="cc558639-1e15-4789-aaae-d6535a4e62e8"
Nov 03 22:45:14.499159 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:14.499143    4431 scope.go:122] "RemoveContainer" containerID="b5a1e8129c605466aa500f815dc3cf73f5f8157377d6e238c700740b06d5a51f"
Nov 03 22:45:14.499411 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:45:14.499383    4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895765-7fxjw_odh-ai-gateway-infra(0d201ce0-fde0-494b-988a-b19d31dbbcda)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895765-7fxjw" podUID="0d201ce0-fde0-494b-988a-b19d31dbbcda"
Nov 03 22:45:14.500864 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:14.500849    4431 scope.go:122] "RemoveContainer" containerID="16b7943d79d69ceb0d567fabf7c1c142eca062a2b409a56befc563e56ca95da6"
Nov 03 22:45:14.501138 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:45:14.501113    4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895765-66lpj_odh-ai-gateway-infra(96676e8f-4e63-42e0-88b3-5588c9c2fed3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895765-66lpj" podUID="96676e8f-4e63-42e0-88b3-5588c9c2fed3"
Nov 03 22:45:14.502541 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:14.502526    4431 scope.go:122] "RemoveContainer" containerID="0b69cb61f3990651913af3b11e451c28c8ff680388cacc95d91f054de062654a"
Nov 03 22:45:14.502917 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:45:14.502824    4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-434ce3-29895765-247jr_odh-ai-gateway-infra(600936df-e54c-429b-ad28-e30a6a76af8f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-434ce3-29895765-247jr" podUID="600936df-e54c-429b-ad28-e30a6a76af8f"
Nov 03 22:45:15.104359 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:15.104299    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-963936-openshift-default-68b65f857c-phdvv"
Nov 03 22:45:15.106850 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:15.106824    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-963936-openshift-default-68b65f857c-phdvv"
Nov 03 22:45:15.969598 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:15.969559    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:45:17.445118 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:17.445078    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc"]
Nov 03 22:45:17.451365 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:17.451340    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc"
Nov 03 22:45:17.470360 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:17.470334    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc"]
Nov 03 22:45:17.476587 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:17.476560    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-50b9b5-openshift-default-dockercfg-n5hxr\""
Nov 03 22:45:17.574190 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:17.574156    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89f34fbd-94ce-4fec-bc0c-38dab6758fa0-istiod-ca-cert\") pod \"e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc\" (UID: \"89f34fbd-94ce-4fec-bc0c-38dab6758fa0\") " pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc"
Nov 03 22:45:17.574391 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:17.574344    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89f34fbd-94ce-4fec-bc0c-38dab6758fa0-istio-podinfo\") pod \"e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc\" (UID: \"89f34fbd-94ce-4fec-bc0c-38dab6758fa0\") " pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc"
Nov 03 22:45:17.574439 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:17.574396    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89f34fbd-94ce-4fec-bc0c-38dab6758fa0-istio-token\") pod \"e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc\" (UID: \"89f34fbd-94ce-4fec-bc0c-38dab6758fa0\") " pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc"
Nov 03 22:45:17.574476 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:17.574443    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qckt\" (UniqueName: \"kubernetes.io/projected/89f34fbd-94ce-4fec-bc0c-38dab6758fa0-kube-api-access-7qckt\") pod \"e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc\" (UID: \"89f34fbd-94ce-4fec-bc0c-38dab6758fa0\") " pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc"
Nov 03 22:45:17.574543 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:17.574519    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89f34fbd-94ce-4fec-bc0c-38dab6758fa0-workload-certs\") pod \"e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc\" (UID: \"89f34fbd-94ce-4fec-bc0c-38dab6758fa0\") " pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc"
Nov 03 22:45:17.574577 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:17.574556    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89f34fbd-94ce-4fec-bc0c-38dab6758fa0-istio-data\") pod \"e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc\" (UID: \"89f34fbd-94ce-4fec-bc0c-38dab6758fa0\") " pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc"
Nov 03 22:45:17.574617 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:17.574605    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89f34fbd-94ce-4fec-bc0c-38dab6758fa0-istio-envoy\") pod \"e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc\" (UID: \"89f34fbd-94ce-4fec-bc0c-38dab6758fa0\") " pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc"
Nov 03 22:45:17.574651 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:17.574625    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89f34fbd-94ce-4fec-bc0c-38dab6758fa0-workload-socket\") pod \"e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc\" (UID: \"89f34fbd-94ce-4fec-bc0c-38dab6758fa0\") " pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc"
Nov 03 22:45:17.574740 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:17.574729    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89f34fbd-94ce-4fec-bc0c-38dab6758fa0-credential-socket\") pod \"e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc\" (UID: \"89f34fbd-94ce-4fec-bc0c-38dab6758fa0\") " pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc"
Nov 03 22:45:17.675933 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:17.675897    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89f34fbd-94ce-4fec-bc0c-38dab6758fa0-credential-socket\") pod \"e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc\" (UID: \"89f34fbd-94ce-4fec-bc0c-38dab6758fa0\") " pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc"
Nov 03 22:45:17.676108 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:17.675978    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89f34fbd-94ce-4fec-bc0c-38dab6758fa0-istiod-ca-cert\") pod \"e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc\" (UID: \"89f34fbd-94ce-4fec-bc0c-38dab6758fa0\") " pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc"
Nov 03 22:45:17.676108 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:17.676081    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89f34fbd-94ce-4fec-bc0c-38dab6758fa0-istio-podinfo\") pod \"e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc\" (UID: \"89f34fbd-94ce-4fec-bc0c-38dab6758fa0\") " pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc"
Nov 03 22:45:17.676375 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:17.676123    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89f34fbd-94ce-4fec-bc0c-38dab6758fa0-istio-token\") pod \"e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc\" (UID: \"89f34fbd-94ce-4fec-bc0c-38dab6758fa0\") " pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc"
Nov 03 22:45:17.676375 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:17.676142    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qckt\" (UniqueName: \"kubernetes.io/projected/89f34fbd-94ce-4fec-bc0c-38dab6758fa0-kube-api-access-7qckt\") pod \"e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc\" (UID: \"89f34fbd-94ce-4fec-bc0c-38dab6758fa0\") " pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc"
Nov 03 22:45:17.676375 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:17.676193    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89f34fbd-94ce-4fec-bc0c-38dab6758fa0-workload-certs\") pod \"e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc\" (UID: \"89f34fbd-94ce-4fec-bc0c-38dab6758fa0\") " pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc"
Nov 03 22:45:17.676375 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:17.676233    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89f34fbd-94ce-4fec-bc0c-38dab6758fa0-istio-data\") pod \"e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc\" (UID: \"89f34fbd-94ce-4fec-bc0c-38dab6758fa0\") " pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc"
Nov 03 22:45:17.676375 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:17.676301    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89f34fbd-94ce-4fec-bc0c-38dab6758fa0-istio-envoy\") pod \"e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc\" (UID: \"89f34fbd-94ce-4fec-bc0c-38dab6758fa0\") " pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc"
Nov 03 22:45:17.676375 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:17.676335    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89f34fbd-94ce-4fec-bc0c-38dab6758fa0-workload-socket\") pod \"e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc\" (UID: \"89f34fbd-94ce-4fec-bc0c-38dab6758fa0\") " pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc"
Nov 03 22:45:17.676375 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:17.676337    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89f34fbd-94ce-4fec-bc0c-38dab6758fa0-credential-socket\") pod \"e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc\" (UID: \"89f34fbd-94ce-4fec-bc0c-38dab6758fa0\") " pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc"
Nov 03 22:45:17.676741 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:17.676650    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89f34fbd-94ce-4fec-bc0c-38dab6758fa0-istio-data\") pod \"e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc\" (UID: \"89f34fbd-94ce-4fec-bc0c-38dab6758fa0\") " pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc"
Nov 03 22:45:17.676741 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:17.676684    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89f34fbd-94ce-4fec-bc0c-38dab6758fa0-workload-socket\") pod \"e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc\" (UID: \"89f34fbd-94ce-4fec-bc0c-38dab6758fa0\") " pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc"
Nov 03 22:45:17.676741 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:17.676708    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89f34fbd-94ce-4fec-bc0c-38dab6758fa0-workload-certs\") pod \"e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc\" (UID: \"89f34fbd-94ce-4fec-bc0c-38dab6758fa0\") " pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc"
Nov 03 22:45:17.676891 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:17.676769    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89f34fbd-94ce-4fec-bc0c-38dab6758fa0-istiod-ca-cert\") pod \"e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc\" (UID: \"89f34fbd-94ce-4fec-bc0c-38dab6758fa0\") " pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc"
Nov 03 22:45:17.678663 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:17.678638    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89f34fbd-94ce-4fec-bc0c-38dab6758fa0-istio-envoy\") pod \"e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc\" (UID: \"89f34fbd-94ce-4fec-bc0c-38dab6758fa0\") " pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc"
Nov 03 22:45:17.678851 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:17.678826    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89f34fbd-94ce-4fec-bc0c-38dab6758fa0-istio-podinfo\") pod \"e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc\" (UID: \"89f34fbd-94ce-4fec-bc0c-38dab6758fa0\") " pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc"
Nov 03 22:45:17.699024 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:17.698963    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qckt\" (UniqueName: \"kubernetes.io/projected/89f34fbd-94ce-4fec-bc0c-38dab6758fa0-kube-api-access-7qckt\") pod \"e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc\" (UID: \"89f34fbd-94ce-4fec-bc0c-38dab6758fa0\") " pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc"
Nov 03 22:45:17.699024 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:17.698972    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89f34fbd-94ce-4fec-bc0c-38dab6758fa0-istio-token\") pod \"e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc\" (UID: \"89f34fbd-94ce-4fec-bc0c-38dab6758fa0\") " pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc"
Nov 03 22:45:17.718506 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:17.718479    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:45:17.767959 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:17.767546    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc"
Nov 03 22:45:18.104721 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:18.104692    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc"]
Nov 03 22:45:18.105059 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:45:18.105033    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89f34fbd_94ce_4fec_bc0c_38dab6758fa0.slice/crio-10410e2f0d8d7bccbf1861223ac1e45e26de5869268438d68ff7532b1557bad0 WatchSource:0}: Error finding container 10410e2f0d8d7bccbf1861223ac1e45e26de5869268438d68ff7532b1557bad0: Status 404 returned error can't find the container with id 10410e2f0d8d7bccbf1861223ac1e45e26de5869268438d68ff7532b1557bad0
Nov 03 22:45:18.108355 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:18.108316    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:45:18.108515 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:18.108397    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:45:18.108515 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:18.108445    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:45:18.536746 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:18.536704    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc" event={"ID":"89f34fbd-94ce-4fec-bc0c-38dab6758fa0","Type":"ContainerStarted","Data":"d2cfaaf94e5ebf5fd3ea00c7418cc3ab881248ea9c7205a4de3163f40108c7f4"}
Nov 03 22:45:18.537195 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:18.536772    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc" event={"ID":"89f34fbd-94ce-4fec-bc0c-38dab6758fa0","Type":"ContainerStarted","Data":"10410e2f0d8d7bccbf1861223ac1e45e26de5869268438d68ff7532b1557bad0"}
Nov 03 22:45:18.596924 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:18.596872    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc" podStartSLOduration=1.5968583299999999 podStartE2EDuration="1.59685833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:18.592728218 +0000 UTC m=+1942.618584079" watchObservedRunningTime="2026-11-03 22:45:18.59685833 +0000 UTC m=+1942.622714194"
Nov 03 22:45:18.771638 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:18.771587    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc"
Nov 03 22:45:19.415415 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:19.415377    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:45:19.769740 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:19.769655    4431 patch_prober.go:29] interesting pod/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Nov 03 22:45:19.770187 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:19.769751    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc" podUID="89f34fbd-94ce-4fec-bc0c-38dab6758fa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Nov 03 22:45:19.885477 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:19.885439    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:45:20.768629 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:20.768595    4431 patch_prober.go:29] interesting pod/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Nov 03 22:45:20.768836 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:20.768650    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc" podUID="89f34fbd-94ce-4fec-bc0c-38dab6758fa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Nov 03 22:45:21.371490 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:21.371457    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:45:21.770611 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:21.770575    4431 patch_prober.go:29] interesting pod/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Nov 03 22:45:21.770884 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:21.770856    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc" podUID="89f34fbd-94ce-4fec-bc0c-38dab6758fa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Nov 03 22:45:22.080455 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:22.080371    4431 scope.go:122] "RemoveContainer" containerID="edb71f93f84f2df5e6cada54258b653e8fdffbe6da81d4d5150552dee5caf97e"
Nov 03 22:45:22.080620 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:22.080542    4431 scope.go:122] "RemoveContainer" containerID="c8b02f4357cadcb5e5967c8d343a98233fc79b58278275af5ff4b518cee8e783"
Nov 03 22:45:22.770542 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:22.770376    4431 patch_prober.go:29] interesting pod/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Nov 03 22:45:22.770542 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:22.770430    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc" podUID="89f34fbd-94ce-4fec-bc0c-38dab6758fa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Nov 03 22:45:23.080574 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:23.080495    4431 scope.go:122] "RemoveContainer" containerID="0d99ffe5900f31df4e6bbcb2cfda9dd24e59c881f88f48e9aa21e170b236f166"
Nov 03 22:45:23.209745 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:23.209715    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:45:23.611937 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:23.611891    4431 generic.go:356] "Generic (PLEG): container finished" podID="f9a74dcb-40e3-415a-a090-3619177efa8a" containerID="4308a7ab3688135100372e5d262d7c43dc3f667ae9f3308a1dca2a16aae8bb09" exitCode=60
Nov 03 22:45:23.612113 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:23.611962    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b1ec3a-29895765-s8js9" event={"ID":"f9a74dcb-40e3-415a-a090-3619177efa8a","Type":"ContainerDied","Data":"4308a7ab3688135100372e5d262d7c43dc3f667ae9f3308a1dca2a16aae8bb09"}
Nov 03 22:45:23.612113 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:23.612017    4431 scope.go:122] "RemoveContainer" containerID="edb71f93f84f2df5e6cada54258b653e8fdffbe6da81d4d5150552dee5caf97e"
Nov 03 22:45:23.612574 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:23.612554    4431 scope.go:122] "RemoveContainer" containerID="4308a7ab3688135100372e5d262d7c43dc3f667ae9f3308a1dca2a16aae8bb09"
Nov 03 22:45:23.613010 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:45:23.612990    4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-b1ec3a-29895765-s8js9_odh-ai-gateway-infra(f9a74dcb-40e3-415a-a090-3619177efa8a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b1ec3a-29895765-s8js9" podUID="f9a74dcb-40e3-415a-a090-3619177efa8a"
Nov 03 22:45:23.614374 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:23.614351    4431 generic.go:356] "Generic (PLEG): container finished" podID="f511dde2-8f9b-4670-bd73-b7ee34d69706" containerID="1b0dd4ada7fe25f8fb5633dc3194be9be466319286c7019a3f0878e000325365" exitCode=60
Nov 03 22:45:23.614538 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:23.614421    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a940d5-29895765-bwkkx" event={"ID":"f511dde2-8f9b-4670-bd73-b7ee34d69706","Type":"ContainerDied","Data":"1b0dd4ada7fe25f8fb5633dc3194be9be466319286c7019a3f0878e000325365"}
Nov 03 22:45:23.614900 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:23.614885    4431 scope.go:122] "RemoveContainer" containerID="1b0dd4ada7fe25f8fb5633dc3194be9be466319286c7019a3f0878e000325365"
Nov 03 22:45:23.615231 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:45:23.615199    4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-a940d5-29895765-bwkkx_odh-ai-gateway-infra(f511dde2-8f9b-4670-bd73-b7ee34d69706)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a940d5-29895765-bwkkx" podUID="f511dde2-8f9b-4670-bd73-b7ee34d69706"
Nov 03 22:45:23.630402 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:23.627532    4431 scope.go:122] "RemoveContainer" containerID="c8b02f4357cadcb5e5967c8d343a98233fc79b58278275af5ff4b518cee8e783"
Nov 03 22:45:23.770243 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:23.769560    4431 patch_prober.go:29] interesting pod/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Nov 03 22:45:23.770243 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:23.769616    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc" podUID="89f34fbd-94ce-4fec-bc0c-38dab6758fa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Nov 03 22:45:24.082327 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:24.082289    4431 scope.go:122] "RemoveContainer" containerID="0b69cb61f3990651913af3b11e451c28c8ff680388cacc95d91f054de062654a"
Nov 03 22:45:24.082779 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:24.082543    4431 scope.go:122] "RemoveContainer" containerID="16b7943d79d69ceb0d567fabf7c1c142eca062a2b409a56befc563e56ca95da6"
Nov 03 22:45:24.632576 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:24.631571    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895765-66lpj" event={"ID":"96676e8f-4e63-42e0-88b3-5588c9c2fed3","Type":"ContainerStarted","Data":"a709568d11d5e6050c5de77cc83b9b2a925ecd53a6d7431ac0fdc30b36403856"}
Nov 03 22:45:24.648326 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:24.648198    4431 generic.go:356] "Generic (PLEG): container finished" podID="cc558639-1e15-4789-aaae-d6535a4e62e8" containerID="f82edc380f5e42414ec819809540c2681df7b2f377ce3e482d44101b4c724ed1" exitCode=60
Nov 03 22:45:24.648326 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:24.648303    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2d619f-29895765-29qhp" event={"ID":"cc558639-1e15-4789-aaae-d6535a4e62e8","Type":"ContainerDied","Data":"f82edc380f5e42414ec819809540c2681df7b2f377ce3e482d44101b4c724ed1"}
Nov 03 22:45:24.648551 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:24.648356    4431 scope.go:122] "RemoveContainer" containerID="0d99ffe5900f31df4e6bbcb2cfda9dd24e59c881f88f48e9aa21e170b236f166"
Nov 03 22:45:24.648875 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:24.648762    4431 scope.go:122] "RemoveContainer" containerID="f82edc380f5e42414ec819809540c2681df7b2f377ce3e482d44101b4c724ed1"
Nov 03 22:45:24.649689 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:45:24.649114    4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-2d619f-29895765-29qhp_odh-ai-gateway-infra(cc558639-1e15-4789-aaae-d6535a4e62e8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2d619f-29895765-29qhp" podUID="cc558639-1e15-4789-aaae-d6535a4e62e8"
Nov 03 22:45:24.667106 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:24.667057    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b1ec3a-29895765-s8js9"]
Nov 03 22:45:24.720050 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:24.719993    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a940d5-29895765-bwkkx"]
Nov 03 22:45:24.897294 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:24.878008    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc"
Nov 03 22:45:24.897294 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:24.878712    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc"
Nov 03 22:45:24.900102 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:24.899594    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc"
Nov 03 22:45:24.913272 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:24.906964    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:45:25.009421 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:25.009388    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b1ec3a-29895765-s8js9"
Nov 03 22:45:25.084056 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:25.084019    4431 scope.go:122] "RemoveContainer" containerID="b5a1e8129c605466aa500f815dc3cf73f5f8157377d6e238c700740b06d5a51f"
Nov 03 22:45:25.084933 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:25.084915    4431 scope.go:122] "RemoveContainer" containerID="11e55cddd05f228273f6ae3c3b12ec29a22c37d2e53844e8574f26ff44a615c0"
Nov 03 22:45:25.153331 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:25.150379    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9a74dcb-40e3-415a-a090-3619177efa8a-kube-api-access-7qhrz\" (UniqueName: \"kubernetes.io/projected/f9a74dcb-40e3-415a-a090-3619177efa8a-kube-api-access-7qhrz\") pod \"f9a74dcb-40e3-415a-a090-3619177efa8a\" (UID: \"f9a74dcb-40e3-415a-a090-3619177efa8a\") "
Nov 03 22:45:25.153331 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:25.150523    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9a74dcb-40e3-415a-a090-3619177efa8a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9a74dcb-40e3-415a-a090-3619177efa8a-maas-api-service-ca\") pod \"f9a74dcb-40e3-415a-a090-3619177efa8a\" (UID: \"f9a74dcb-40e3-415a-a090-3619177efa8a\") "
Nov 03 22:45:25.153331 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:25.153036    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9a74dcb-40e3-415a-a090-3619177efa8a-maas-api-service-ca" pod "f9a74dcb-40e3-415a-a090-3619177efa8a" (UID: "f9a74dcb-40e3-415a-a090-3619177efa8a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:45:25.154138 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:25.154109    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9a74dcb-40e3-415a-a090-3619177efa8a-kube-api-access-7qhrz" pod "f9a74dcb-40e3-415a-a090-3619177efa8a" (UID: "f9a74dcb-40e3-415a-a090-3619177efa8a"). InnerVolumeSpecName "kube-api-access-7qhrz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:45:25.155208 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:25.155190    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a940d5-29895765-bwkkx"
Nov 03 22:45:25.256355 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:25.253418    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f511dde2-8f9b-4670-bd73-b7ee34d69706-kube-api-access-2gdzq\" (UniqueName: \"kubernetes.io/projected/f511dde2-8f9b-4670-bd73-b7ee34d69706-kube-api-access-2gdzq\") pod \"f511dde2-8f9b-4670-bd73-b7ee34d69706\" (UID: \"f511dde2-8f9b-4670-bd73-b7ee34d69706\") "
Nov 03 22:45:25.256355 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:25.253780    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f511dde2-8f9b-4670-bd73-b7ee34d69706-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f511dde2-8f9b-4670-bd73-b7ee34d69706-maas-api-service-ca\") pod \"f511dde2-8f9b-4670-bd73-b7ee34d69706\" (UID: \"f511dde2-8f9b-4670-bd73-b7ee34d69706\") "
Nov 03 22:45:25.256355 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:25.254367    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f511dde2-8f9b-4670-bd73-b7ee34d69706-maas-api-service-ca" pod "f511dde2-8f9b-4670-bd73-b7ee34d69706" (UID: "f511dde2-8f9b-4670-bd73-b7ee34d69706"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:45:25.256355 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:25.254597    4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f511dde2-8f9b-4670-bd73-b7ee34d69706-maas-api-service-ca\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:45:25.256355 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:25.254617    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qhrz\" (UniqueName: \"kubernetes.io/projected/f9a74dcb-40e3-415a-a090-3619177efa8a-kube-api-access-7qhrz\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:45:25.256355 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:25.254634    4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9a74dcb-40e3-415a-a090-3619177efa8a-maas-api-service-ca\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:45:25.257080 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:25.256916    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f511dde2-8f9b-4670-bd73-b7ee34d69706-kube-api-access-2gdzq" pod "f511dde2-8f9b-4670-bd73-b7ee34d69706" (UID: "f511dde2-8f9b-4670-bd73-b7ee34d69706"). InnerVolumeSpecName "kube-api-access-2gdzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:45:25.356553 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:25.356522    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gdzq\" (UniqueName: \"kubernetes.io/projected/f511dde2-8f9b-4670-bd73-b7ee34d69706-kube-api-access-2gdzq\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:45:25.660462 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:25.660431    4431 generic.go:356] "Generic (PLEG): container finished" podID="96676e8f-4e63-42e0-88b3-5588c9c2fed3" containerID="a709568d11d5e6050c5de77cc83b9b2a925ecd53a6d7431ac0fdc30b36403856" exitCode=60
Nov 03 22:45:25.660594 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:25.660505    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895765-66lpj" event={"ID":"96676e8f-4e63-42e0-88b3-5588c9c2fed3","Type":"ContainerDied","Data":"a709568d11d5e6050c5de77cc83b9b2a925ecd53a6d7431ac0fdc30b36403856"}
Nov 03 22:45:25.660594 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:25.660579    4431 scope.go:122] "RemoveContainer" containerID="16b7943d79d69ceb0d567fabf7c1c142eca062a2b409a56befc563e56ca95da6"
Nov 03 22:45:25.660989 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:25.660968    4431 scope.go:122] "RemoveContainer" containerID="a709568d11d5e6050c5de77cc83b9b2a925ecd53a6d7431ac0fdc30b36403856"
Nov 03 22:45:25.661456 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:45:25.661395    4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895765-66lpj_odh-ai-gateway-infra(96676e8f-4e63-42e0-88b3-5588c9c2fed3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895765-66lpj" podUID="96676e8f-4e63-42e0-88b3-5588c9c2fed3"
Nov 03 22:45:25.663518 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:25.663498    4431 generic.go:356] "Generic (PLEG): container finished" podID="600936df-e54c-429b-ad28-e30a6a76af8f" containerID="4bd4d8e4ec82e0905ad5c575150b00b9b24bf6b6ec8c95969b5cd94320d498d9" exitCode=60
Nov 03 22:45:25.663606 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:25.663566    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-434ce3-29895765-247jr" event={"ID":"600936df-e54c-429b-ad28-e30a6a76af8f","Type":"ContainerDied","Data":"4bd4d8e4ec82e0905ad5c575150b00b9b24bf6b6ec8c95969b5cd94320d498d9"}
Nov 03 22:45:25.663940 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:25.663919    4431 scope.go:122] "RemoveContainer" containerID="4bd4d8e4ec82e0905ad5c575150b00b9b24bf6b6ec8c95969b5cd94320d498d9"
Nov 03 22:45:25.664185 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:45:25.664164    4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-434ce3-29895765-247jr_odh-ai-gateway-infra(600936df-e54c-429b-ad28-e30a6a76af8f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-434ce3-29895765-247jr" podUID="600936df-e54c-429b-ad28-e30a6a76af8f"
Nov 03 22:45:25.665423 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:25.665400    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0a7065-29895765-2x2gq" event={"ID":"7ef57864-5184-4b21-8fae-201006734abf","Type":"ContainerStarted","Data":"39d245368843cb0ec7513af03b77fd0c67777e7469a7caf0fe1909f0e675ed6f"}
Nov 03 22:45:25.666954 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:25.666938    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b1ec3a-29895765-s8js9"
Nov 03 22:45:25.667037 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:25.666987    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b1ec3a-29895765-s8js9" event={"ID":"f9a74dcb-40e3-415a-a090-3619177efa8a","Type":"ContainerDied","Data":"93286abb12224f62887f6817bdc542829f06c82e86614dfe2b5e10cb9e3d8f7b"}
Nov 03 22:45:25.670238 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:25.670206    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a940d5-29895765-bwkkx"
Nov 03 22:45:25.670327 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:25.670247    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a940d5-29895765-bwkkx" event={"ID":"f511dde2-8f9b-4670-bd73-b7ee34d69706","Type":"ContainerDied","Data":"05a4eb588a17ccf92c4ac026c884a146ac26aaab8ef1089ec0d004a3764e2789"}
Nov 03 22:45:25.675295 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:25.675276    4431 scope.go:122] "RemoveContainer" containerID="0b69cb61f3990651913af3b11e451c28c8ff680388cacc95d91f054de062654a"
Nov 03 22:45:25.686158 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:25.686133    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895765-66lpj"]
Nov 03 22:45:25.690614 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:25.690563    4431 scope.go:122] "RemoveContainer" containerID="4308a7ab3688135100372e5d262d7c43dc3f667ae9f3308a1dca2a16aae8bb09"
Nov 03 22:45:25.700254 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:25.700228    4431 scope.go:122] "RemoveContainer" containerID="1b0dd4ada7fe25f8fb5633dc3194be9be466319286c7019a3f0878e000325365"
Nov 03 22:45:25.745609 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:25.745575    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2d619f-29895765-29qhp"]
Nov 03 22:45:25.784635 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:25.784598    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a940d5-29895765-bwkkx"]
Nov 03 22:45:25.789599 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:25.789569    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a940d5-29895765-bwkkx"]
Nov 03 22:45:25.868787 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:25.868718    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b1ec3a-29895765-s8js9"]
Nov 03 22:45:25.872798 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:25.872765    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b1ec3a-29895765-s8js9"]
Nov 03 22:45:25.902609 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:25.902587    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2d619f-29895765-29qhp"
Nov 03 22:45:25.968875 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:25.968790    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc558639-1e15-4789-aaae-d6535a4e62e8-kube-api-access-dfml7\" (UniqueName: \"kubernetes.io/projected/cc558639-1e15-4789-aaae-d6535a4e62e8-kube-api-access-dfml7\") pod \"cc558639-1e15-4789-aaae-d6535a4e62e8\" (UID: \"cc558639-1e15-4789-aaae-d6535a4e62e8\") "
Nov 03 22:45:25.969055 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:25.969045    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc558639-1e15-4789-aaae-d6535a4e62e8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc558639-1e15-4789-aaae-d6535a4e62e8-maas-api-service-ca\") pod \"cc558639-1e15-4789-aaae-d6535a4e62e8\" (UID: \"cc558639-1e15-4789-aaae-d6535a4e62e8\") "
Nov 03 22:45:25.969521 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:25.969482    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc558639-1e15-4789-aaae-d6535a4e62e8-maas-api-service-ca" pod "cc558639-1e15-4789-aaae-d6535a4e62e8" (UID: "cc558639-1e15-4789-aaae-d6535a4e62e8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:45:25.969726 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:25.969707    4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc558639-1e15-4789-aaae-d6535a4e62e8-maas-api-service-ca\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:45:25.971606 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:25.971571    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc558639-1e15-4789-aaae-d6535a4e62e8-kube-api-access-dfml7" pod "cc558639-1e15-4789-aaae-d6535a4e62e8" (UID: "cc558639-1e15-4789-aaae-d6535a4e62e8"). InnerVolumeSpecName "kube-api-access-dfml7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:45:26.071019 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:26.070981    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfml7\" (UniqueName: \"kubernetes.io/projected/cc558639-1e15-4789-aaae-d6535a4e62e8-kube-api-access-dfml7\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:45:26.085988 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:26.085962    4431 scope.go:122] "RemoveContainer" containerID="1ad647742f262cc608eee961cf59f4b4d578e800917ca2f3361076563dde9278"
Nov 03 22:45:26.086455 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:26.086148    4431 scope.go:122] "RemoveContainer" containerID="3f58bda8f9ae34e5a4da9bde0f08c7f9607bab226a8f6082d2fd9d2bae71493a"
Nov 03 22:45:26.086455 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:26.086307    4431 scope.go:122] "RemoveContainer" containerID="e84fbcea49eaa20eadc6ca58936e17ea2c3a68ea2bcf33ca507ee386fb7f884e"
Nov 03 22:45:26.106370 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:26.106344    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f511dde2-8f9b-4670-bd73-b7ee34d69706" path="/var/lib/kubelet/pods/f511dde2-8f9b-4670-bd73-b7ee34d69706/volumes"
Nov 03 22:45:26.106996 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:26.106973    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9a74dcb-40e3-415a-a090-3619177efa8a" path="/var/lib/kubelet/pods/f9a74dcb-40e3-415a-a090-3619177efa8a/volumes"
Nov 03 22:45:26.437794 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:26.437761    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:45:26.681786 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:26.681752    4431 generic.go:356] "Generic (PLEG): container finished" podID="7ef57864-5184-4b21-8fae-201006734abf" containerID="39d245368843cb0ec7513af03b77fd0c67777e7469a7caf0fe1909f0e675ed6f" exitCode=60
Nov 03 22:45:26.681935 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:26.681824    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0a7065-29895765-2x2gq" event={"ID":"7ef57864-5184-4b21-8fae-201006734abf","Type":"ContainerDied","Data":"39d245368843cb0ec7513af03b77fd0c67777e7469a7caf0fe1909f0e675ed6f"}
Nov 03 22:45:26.681935 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:26.681883    4431 scope.go:122] "RemoveContainer" containerID="11e55cddd05f228273f6ae3c3b12ec29a22c37d2e53844e8574f26ff44a615c0"
Nov 03 22:45:26.682374 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:26.682350    4431 scope.go:122] "RemoveContainer" containerID="39d245368843cb0ec7513af03b77fd0c67777e7469a7caf0fe1909f0e675ed6f"
Nov 03 22:45:26.682889 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:45:26.682814    4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-0a7065-29895765-2x2gq_odh-ai-gateway-infra(7ef57864-5184-4b21-8fae-201006734abf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0a7065-29895765-2x2gq" podUID="7ef57864-5184-4b21-8fae-201006734abf"
Nov 03 22:45:26.684113 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:26.684088    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7940ea-29895765-nvrxh" event={"ID":"4b6f92d6-3fbb-48fe-854f-6b5b899f56d1","Type":"ContainerStarted","Data":"ddb759ca587608a9d63a252de252ce3cf89790f8565a96cafc3dada30dc1e2ca"}
Nov 03 22:45:26.686696 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:26.686667    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2d619f-29895765-29qhp" event={"ID":"cc558639-1e15-4789-aaae-d6535a4e62e8","Type":"ContainerDied","Data":"1b0d1abc4afb42a23c9f9c561d8437e1605708d1ef928a193bb74be296207294"}
Nov 03 22:45:26.686696 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:26.686681    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2d619f-29895765-29qhp"
Nov 03 22:45:26.688665 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:26.688636    4431 generic.go:356] "Generic (PLEG): container finished" podID="0d201ce0-fde0-494b-988a-b19d31dbbcda" containerID="f0b6ac6553aee8267823cfd6506fdc5ff20e787d759b3487fb743ea378ab5569" exitCode=60
Nov 03 22:45:26.688760 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:26.688706    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895765-7fxjw" event={"ID":"0d201ce0-fde0-494b-988a-b19d31dbbcda","Type":"ContainerDied","Data":"f0b6ac6553aee8267823cfd6506fdc5ff20e787d759b3487fb743ea378ab5569"}
Nov 03 22:45:26.689152 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:26.689131    4431 scope.go:122] "RemoveContainer" containerID="f0b6ac6553aee8267823cfd6506fdc5ff20e787d759b3487fb743ea378ab5569"
Nov 03 22:45:26.689572 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:45:26.689547    4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895765-7fxjw_odh-ai-gateway-infra(0d201ce0-fde0-494b-988a-b19d31dbbcda)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895765-7fxjw" podUID="0d201ce0-fde0-494b-988a-b19d31dbbcda"
Nov 03 22:45:26.697127 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:26.697109    4431 scope.go:122] "RemoveContainer" containerID="f82edc380f5e42414ec819809540c2681df7b2f377ce3e482d44101b4c724ed1"
Nov 03 22:45:26.710556 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:26.710530    4431 scope.go:122] "RemoveContainer" containerID="b5a1e8129c605466aa500f815dc3cf73f5f8157377d6e238c700740b06d5a51f"
Nov 03 22:45:26.758728 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:26.758700    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0a7065-29895765-2x2gq"]
Nov 03 22:45:26.766371 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:26.765454    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2d619f-29895765-29qhp"]
Nov 03 22:45:26.785874 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:26.785839    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2d619f-29895765-29qhp"]
Nov 03 22:45:26.837652 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:26.837570    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-434ce3-29895765-247jr"]
Nov 03 22:45:26.890791 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:26.890762    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895765-66lpj"
Nov 03 22:45:27.006189 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:27.006141    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96676e8f-4e63-42e0-88b3-5588c9c2fed3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96676e8f-4e63-42e0-88b3-5588c9c2fed3-maas-api-service-ca\") pod \"96676e8f-4e63-42e0-88b3-5588c9c2fed3\" (UID: \"96676e8f-4e63-42e0-88b3-5588c9c2fed3\") "
Nov 03 22:45:27.006400 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:27.006260    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96676e8f-4e63-42e0-88b3-5588c9c2fed3-kube-api-access-kbvpc\" (UniqueName: \"kubernetes.io/projected/96676e8f-4e63-42e0-88b3-5588c9c2fed3-kube-api-access-kbvpc\") pod \"96676e8f-4e63-42e0-88b3-5588c9c2fed3\" (UID: \"96676e8f-4e63-42e0-88b3-5588c9c2fed3\") "
Nov 03 22:45:27.006580 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:27.006554    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96676e8f-4e63-42e0-88b3-5588c9c2fed3-maas-api-service-ca" pod "96676e8f-4e63-42e0-88b3-5588c9c2fed3" (UID: "96676e8f-4e63-42e0-88b3-5588c9c2fed3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:45:27.007163 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:27.007104    4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96676e8f-4e63-42e0-88b3-5588c9c2fed3-maas-api-service-ca\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:45:27.009540 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:27.009513    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96676e8f-4e63-42e0-88b3-5588c9c2fed3-kube-api-access-kbvpc" pod "96676e8f-4e63-42e0-88b3-5588c9c2fed3" (UID: "96676e8f-4e63-42e0-88b3-5588c9c2fed3"). InnerVolumeSpecName "kube-api-access-kbvpc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:45:27.009957 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:27.009939    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-434ce3-29895765-247jr"
Nov 03 22:45:27.109067 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:27.108977    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/600936df-e54c-429b-ad28-e30a6a76af8f-kube-api-access-pmzhd\" (UniqueName: \"kubernetes.io/projected/600936df-e54c-429b-ad28-e30a6a76af8f-kube-api-access-pmzhd\") pod \"600936df-e54c-429b-ad28-e30a6a76af8f\" (UID: \"600936df-e54c-429b-ad28-e30a6a76af8f\") "
Nov 03 22:45:27.109572 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:27.109170    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/600936df-e54c-429b-ad28-e30a6a76af8f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/600936df-e54c-429b-ad28-e30a6a76af8f-maas-api-service-ca\") pod \"600936df-e54c-429b-ad28-e30a6a76af8f\" (UID: \"600936df-e54c-429b-ad28-e30a6a76af8f\") "
Nov 03 22:45:27.109660 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:27.109605    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/600936df-e54c-429b-ad28-e30a6a76af8f-maas-api-service-ca" pod "600936df-e54c-429b-ad28-e30a6a76af8f" (UID: "600936df-e54c-429b-ad28-e30a6a76af8f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:45:27.109944 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:27.109923    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbvpc\" (UniqueName: \"kubernetes.io/projected/96676e8f-4e63-42e0-88b3-5588c9c2fed3-kube-api-access-kbvpc\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:45:27.110012 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:27.109948    4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/600936df-e54c-429b-ad28-e30a6a76af8f-maas-api-service-ca\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:45:27.111630 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:27.111604    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/600936df-e54c-429b-ad28-e30a6a76af8f-kube-api-access-pmzhd" pod "600936df-e54c-429b-ad28-e30a6a76af8f" (UID: "600936df-e54c-429b-ad28-e30a6a76af8f"). InnerVolumeSpecName "kube-api-access-pmzhd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:45:27.211984 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:27.211951    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmzhd\" (UniqueName: \"kubernetes.io/projected/600936df-e54c-429b-ad28-e30a6a76af8f-kube-api-access-pmzhd\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:45:27.704277 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:27.704245    4431 generic.go:356] "Generic (PLEG): container finished" podID="57ada6e6-75cf-47a4-87d6-8ad44c0e442e" containerID="52434fb2cd5fcc0a93ac07c6b06f0ab1f886c53e5140c594ce7663f34ae08030" exitCode=60
Nov 03 22:45:27.704463 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:27.704332    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-963936-29895765-52lrg" event={"ID":"57ada6e6-75cf-47a4-87d6-8ad44c0e442e","Type":"ContainerDied","Data":"52434fb2cd5fcc0a93ac07c6b06f0ab1f886c53e5140c594ce7663f34ae08030"}
Nov 03 22:45:27.704463 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:27.704402    4431 scope.go:122] "RemoveContainer" containerID="1ad647742f262cc608eee961cf59f4b4d578e800917ca2f3361076563dde9278"
Nov 03 22:45:27.704917 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:27.704889    4431 scope.go:122] "RemoveContainer" containerID="52434fb2cd5fcc0a93ac07c6b06f0ab1f886c53e5140c594ce7663f34ae08030"
Nov 03 22:45:27.705690 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:45:27.705349    4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-963936-29895765-52lrg_odh-ai-gateway-infra(57ada6e6-75cf-47a4-87d6-8ad44c0e442e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-963936-29895765-52lrg" podUID="57ada6e6-75cf-47a4-87d6-8ad44c0e442e"
Nov 03 22:45:27.705951 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:27.705926    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895765-66lpj" event={"ID":"96676e8f-4e63-42e0-88b3-5588c9c2fed3","Type":"ContainerDied","Data":"61dd653b4d555bf27b98538225f6b23f3bc086c2c5ccf2b9e0d91cbd07bc785f"}
Nov 03 22:45:27.706027 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:27.705937    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895765-66lpj"
Nov 03 22:45:27.707337 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:27.707319    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-434ce3-29895765-247jr"
Nov 03 22:45:27.707412 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:27.707332    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-434ce3-29895765-247jr" event={"ID":"600936df-e54c-429b-ad28-e30a6a76af8f","Type":"ContainerDied","Data":"8c730eb8aaa519202d3923a3312c69f154bb3e40a1f94e6aff898ae8fbf4028e"}
Nov 03 22:45:27.720194 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:27.720170    4431 generic.go:356] "Generic (PLEG): container finished" podID="8b5f2047-8624-42fb-9881-6d81c25867ee" containerID="25a91d81884ac6da31c07b09c71204e96c06e917d5910f71c8ca64b422a1eaca" exitCode=60
Nov 03 22:45:27.720319 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:27.720261    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2d6076-29895765-5njwc" event={"ID":"8b5f2047-8624-42fb-9881-6d81c25867ee","Type":"ContainerDied","Data":"25a91d81884ac6da31c07b09c71204e96c06e917d5910f71c8ca64b422a1eaca"}
Nov 03 22:45:27.720495 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:27.720469    4431 scope.go:122] "RemoveContainer" containerID="a709568d11d5e6050c5de77cc83b9b2a925ecd53a6d7431ac0fdc30b36403856"
Nov 03 22:45:27.720678 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:27.720662    4431 scope.go:122] "RemoveContainer" containerID="25a91d81884ac6da31c07b09c71204e96c06e917d5910f71c8ca64b422a1eaca"
Nov 03 22:45:27.721007 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:45:27.720987    4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-2d6076-29895765-5njwc_odh-ai-gateway-infra(8b5f2047-8624-42fb-9881-6d81c25867ee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2d6076-29895765-5njwc" podUID="8b5f2047-8624-42fb-9881-6d81c25867ee"
Nov 03 22:45:27.723078 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:27.723055    4431 generic.go:356] "Generic (PLEG): container finished" podID="4b6f92d6-3fbb-48fe-854f-6b5b899f56d1" containerID="ddb759ca587608a9d63a252de252ce3cf89790f8565a96cafc3dada30dc1e2ca" exitCode=60
Nov 03 22:45:27.723182 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:27.723123    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7940ea-29895765-nvrxh" event={"ID":"4b6f92d6-3fbb-48fe-854f-6b5b899f56d1","Type":"ContainerDied","Data":"ddb759ca587608a9d63a252de252ce3cf89790f8565a96cafc3dada30dc1e2ca"}
Nov 03 22:45:27.723549 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:27.723532    4431 scope.go:122] "RemoveContainer" containerID="ddb759ca587608a9d63a252de252ce3cf89790f8565a96cafc3dada30dc1e2ca"
Nov 03 22:45:27.723855 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:45:27.723836    4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-7940ea-29895765-nvrxh_odh-ai-gateway-infra(4b6f92d6-3fbb-48fe-854f-6b5b899f56d1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7940ea-29895765-nvrxh" podUID="4b6f92d6-3fbb-48fe-854f-6b5b899f56d1"
Nov 03 22:45:27.733070 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:27.733047    4431 scope.go:122] "RemoveContainer" containerID="4bd4d8e4ec82e0905ad5c575150b00b9b24bf6b6ec8c95969b5cd94320d498d9"
Nov 03 22:45:27.746909 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:27.746855    4431 scope.go:122] "RemoveContainer" containerID="e84fbcea49eaa20eadc6ca58936e17ea2c3a68ea2bcf33ca507ee386fb7f884e"
Nov 03 22:45:27.769167 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:27.769146    4431 scope.go:122] "RemoveContainer" containerID="3f58bda8f9ae34e5a4da9bde0f08c7f9607bab226a8f6082d2fd9d2bae71493a"
Nov 03 22:45:27.824706 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:27.824670    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-434ce3-29895765-247jr"]
Nov 03 22:45:27.831742 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:27.831708    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-434ce3-29895765-247jr"]
Nov 03 22:45:27.868821 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:27.868789    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895765-66lpj"]
Nov 03 22:45:27.875609 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:27.875581    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895765-66lpj"]
Nov 03 22:45:27.885092 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:27.885062    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7940ea-29895765-nvrxh"]
Nov 03 22:45:27.911458 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:27.911432    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0a7065-29895765-2x2gq"
Nov 03 22:45:27.914703 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:27.914680    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895765-7fxjw"]
Nov 03 22:45:28.025728 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:28.021868    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ef57864-5184-4b21-8fae-201006734abf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ef57864-5184-4b21-8fae-201006734abf-maas-api-service-ca\") pod \"7ef57864-5184-4b21-8fae-201006734abf\" (UID: \"7ef57864-5184-4b21-8fae-201006734abf\") "
Nov 03 22:45:28.025728 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:28.022207    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ef57864-5184-4b21-8fae-201006734abf-kube-api-access-k72qh\" (UniqueName: \"kubernetes.io/projected/7ef57864-5184-4b21-8fae-201006734abf-kube-api-access-k72qh\") pod \"7ef57864-5184-4b21-8fae-201006734abf\" (UID: \"7ef57864-5184-4b21-8fae-201006734abf\") "
Nov 03 22:45:28.025728 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:28.022395    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ef57864-5184-4b21-8fae-201006734abf-maas-api-service-ca" pod "7ef57864-5184-4b21-8fae-201006734abf" (UID: "7ef57864-5184-4b21-8fae-201006734abf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:45:28.025728 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:28.022944    4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ef57864-5184-4b21-8fae-201006734abf-maas-api-service-ca\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:45:28.025728 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:28.025285    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ef57864-5184-4b21-8fae-201006734abf-kube-api-access-k72qh" pod "7ef57864-5184-4b21-8fae-201006734abf" (UID: "7ef57864-5184-4b21-8fae-201006734abf"). InnerVolumeSpecName "kube-api-access-k72qh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:45:28.081957 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:28.081929    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895765-7fxjw"
Nov 03 22:45:28.093113 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:28.093085    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="600936df-e54c-429b-ad28-e30a6a76af8f" path="/var/lib/kubelet/pods/600936df-e54c-429b-ad28-e30a6a76af8f/volumes"
Nov 03 22:45:28.093720 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:28.093702    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96676e8f-4e63-42e0-88b3-5588c9c2fed3" path="/var/lib/kubelet/pods/96676e8f-4e63-42e0-88b3-5588c9c2fed3/volumes"
Nov 03 22:45:28.094149 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:28.094136    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc558639-1e15-4789-aaae-d6535a4e62e8" path="/var/lib/kubelet/pods/cc558639-1e15-4789-aaae-d6535a4e62e8/volumes"
Nov 03 22:45:28.128649 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:28.128612    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k72qh\" (UniqueName: \"kubernetes.io/projected/7ef57864-5184-4b21-8fae-201006734abf-kube-api-access-k72qh\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:45:28.165839 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:28.159956    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:45:28.230373 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:28.230282    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d201ce0-fde0-494b-988a-b19d31dbbcda-kube-api-access-csp9s\" (UniqueName: \"kubernetes.io/projected/0d201ce0-fde0-494b-988a-b19d31dbbcda-kube-api-access-csp9s\") pod \"0d201ce0-fde0-494b-988a-b19d31dbbcda\" (UID: \"0d201ce0-fde0-494b-988a-b19d31dbbcda\") "
Nov 03 22:45:28.230373 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:28.230329    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d201ce0-fde0-494b-988a-b19d31dbbcda-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d201ce0-fde0-494b-988a-b19d31dbbcda-maas-api-service-ca\") pod \"0d201ce0-fde0-494b-988a-b19d31dbbcda\" (UID: \"0d201ce0-fde0-494b-988a-b19d31dbbcda\") "
Nov 03 22:45:28.232060 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:28.232014    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d201ce0-fde0-494b-988a-b19d31dbbcda-maas-api-service-ca" pod "0d201ce0-fde0-494b-988a-b19d31dbbcda" (UID: "0d201ce0-fde0-494b-988a-b19d31dbbcda"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:45:28.233279 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:28.233248    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d201ce0-fde0-494b-988a-b19d31dbbcda-kube-api-access-csp9s" pod "0d201ce0-fde0-494b-988a-b19d31dbbcda" (UID: "0d201ce0-fde0-494b-988a-b19d31dbbcda"). InnerVolumeSpecName "kube-api-access-csp9s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:45:28.332381 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:28.332347    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csp9s\" (UniqueName: \"kubernetes.io/projected/0d201ce0-fde0-494b-988a-b19d31dbbcda-kube-api-access-csp9s\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:45:28.332381 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:28.332377    4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d201ce0-fde0-494b-988a-b19d31dbbcda-maas-api-service-ca\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:45:28.736735 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:28.736703    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895765-7fxjw"
Nov 03 22:45:28.736735 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:28.736720    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895765-7fxjw" event={"ID":"0d201ce0-fde0-494b-988a-b19d31dbbcda","Type":"ContainerDied","Data":"580100edab54278fe7728359395c74714dcff381ad6ea8da768f7a229f5edfad"}
Nov 03 22:45:28.736968 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:28.736833    4431 scope.go:122] "RemoveContainer" containerID="f0b6ac6553aee8267823cfd6506fdc5ff20e787d759b3487fb743ea378ab5569"
Nov 03 22:45:28.742856 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:28.742834    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0a7065-29895765-2x2gq"
Nov 03 22:45:28.742999 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:28.742892    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0a7065-29895765-2x2gq" event={"ID":"7ef57864-5184-4b21-8fae-201006734abf","Type":"ContainerDied","Data":"e16beff988df659af008188c5218b5a0b833739f87a7f5c6fd02df6b965b2282"}
Nov 03 22:45:28.751320 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:28.751299    4431 scope.go:122] "RemoveContainer" containerID="39d245368843cb0ec7513af03b77fd0c67777e7469a7caf0fe1909f0e675ed6f"
Nov 03 22:45:28.757346 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:28.757323    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-963936-29895765-52lrg"]
Nov 03 22:45:28.803133 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:28.803102    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0a7065-29895765-2x2gq"]
Nov 03 22:45:28.809713 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:28.809656    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0a7065-29895765-2x2gq"]
Nov 03 22:45:28.920172 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:28.920143    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895765-7fxjw"]
Nov 03 22:45:28.927874 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:28.927847    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895765-7fxjw"]
Nov 03 22:45:28.959334 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:28.959296    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7940ea-29895765-nvrxh"
Nov 03 22:45:28.969591 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:28.969562    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2d6076-29895765-5njwc"]
Nov 03 22:45:28.996816 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:28.996742    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-963936-29895765-52lrg"
Nov 03 22:45:29.043261 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:29.043202    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b6f92d6-3fbb-48fe-854f-6b5b899f56d1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b6f92d6-3fbb-48fe-854f-6b5b899f56d1-maas-api-service-ca\") pod \"4b6f92d6-3fbb-48fe-854f-6b5b899f56d1\" (UID: \"4b6f92d6-3fbb-48fe-854f-6b5b899f56d1\") "
Nov 03 22:45:29.043587 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:29.043563    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b6f92d6-3fbb-48fe-854f-6b5b899f56d1-kube-api-access-h8pqq\" (UniqueName: \"kubernetes.io/projected/4b6f92d6-3fbb-48fe-854f-6b5b899f56d1-kube-api-access-h8pqq\") pod \"4b6f92d6-3fbb-48fe-854f-6b5b899f56d1\" (UID: \"4b6f92d6-3fbb-48fe-854f-6b5b899f56d1\") "
Nov 03 22:45:29.043987 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:29.043581    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b6f92d6-3fbb-48fe-854f-6b5b899f56d1-maas-api-service-ca" pod "4b6f92d6-3fbb-48fe-854f-6b5b899f56d1" (UID: "4b6f92d6-3fbb-48fe-854f-6b5b899f56d1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:45:29.049762 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:29.049712    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b6f92d6-3fbb-48fe-854f-6b5b899f56d1-kube-api-access-h8pqq" pod "4b6f92d6-3fbb-48fe-854f-6b5b899f56d1" (UID: "4b6f92d6-3fbb-48fe-854f-6b5b899f56d1"). InnerVolumeSpecName "kube-api-access-h8pqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:45:29.145738 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:29.145709    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57ada6e6-75cf-47a4-87d6-8ad44c0e442e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57ada6e6-75cf-47a4-87d6-8ad44c0e442e-maas-api-service-ca\") pod \"57ada6e6-75cf-47a4-87d6-8ad44c0e442e\" (UID: \"57ada6e6-75cf-47a4-87d6-8ad44c0e442e\") "
Nov 03 22:45:29.146207 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:29.145752    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57ada6e6-75cf-47a4-87d6-8ad44c0e442e-kube-api-access-hv46j\" (UniqueName: \"kubernetes.io/projected/57ada6e6-75cf-47a4-87d6-8ad44c0e442e-kube-api-access-hv46j\") pod \"57ada6e6-75cf-47a4-87d6-8ad44c0e442e\" (UID: \"57ada6e6-75cf-47a4-87d6-8ad44c0e442e\") "
Nov 03 22:45:29.146207 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:29.146096    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57ada6e6-75cf-47a4-87d6-8ad44c0e442e-maas-api-service-ca" pod "57ada6e6-75cf-47a4-87d6-8ad44c0e442e" (UID: "57ada6e6-75cf-47a4-87d6-8ad44c0e442e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:45:29.146639 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:29.146615    4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57ada6e6-75cf-47a4-87d6-8ad44c0e442e-maas-api-service-ca\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:45:29.146730 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:29.146642    4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b6f92d6-3fbb-48fe-854f-6b5b899f56d1-maas-api-service-ca\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:45:29.146730 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:29.146652    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8pqq\" (UniqueName: \"kubernetes.io/projected/4b6f92d6-3fbb-48fe-854f-6b5b899f56d1-kube-api-access-h8pqq\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:45:29.148893 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:29.148870    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57ada6e6-75cf-47a4-87d6-8ad44c0e442e-kube-api-access-hv46j" pod "57ada6e6-75cf-47a4-87d6-8ad44c0e442e" (UID: "57ada6e6-75cf-47a4-87d6-8ad44c0e442e"). InnerVolumeSpecName "kube-api-access-hv46j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:45:29.191208 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:29.191186    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2d6076-29895765-5njwc"
Nov 03 22:45:29.249022 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:29.248932    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hv46j\" (UniqueName: \"kubernetes.io/projected/57ada6e6-75cf-47a4-87d6-8ad44c0e442e-kube-api-access-hv46j\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:45:29.349554 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:29.349523    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b5f2047-8624-42fb-9881-6d81c25867ee-kube-api-access-r7kcp\" (UniqueName: \"kubernetes.io/projected/8b5f2047-8624-42fb-9881-6d81c25867ee-kube-api-access-r7kcp\") pod \"8b5f2047-8624-42fb-9881-6d81c25867ee\" (UID: \"8b5f2047-8624-42fb-9881-6d81c25867ee\") "
Nov 03 22:45:29.349733 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:29.349669    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b5f2047-8624-42fb-9881-6d81c25867ee-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b5f2047-8624-42fb-9881-6d81c25867ee-maas-api-service-ca\") pod \"8b5f2047-8624-42fb-9881-6d81c25867ee\" (UID: \"8b5f2047-8624-42fb-9881-6d81c25867ee\") "
Nov 03 22:45:29.350106 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:29.350073    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b5f2047-8624-42fb-9881-6d81c25867ee-maas-api-service-ca" pod "8b5f2047-8624-42fb-9881-6d81c25867ee" (UID: "8b5f2047-8624-42fb-9881-6d81c25867ee"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:45:29.350416 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:29.350394    4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b5f2047-8624-42fb-9881-6d81c25867ee-maas-api-service-ca\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:45:29.351756 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:29.351733    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b5f2047-8624-42fb-9881-6d81c25867ee-kube-api-access-r7kcp" pod "8b5f2047-8624-42fb-9881-6d81c25867ee" (UID: "8b5f2047-8624-42fb-9881-6d81c25867ee"). InnerVolumeSpecName "kube-api-access-r7kcp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:45:29.452162 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:29.452130    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7kcp\" (UniqueName: \"kubernetes.io/projected/8b5f2047-8624-42fb-9881-6d81c25867ee-kube-api-access-r7kcp\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:45:29.759786 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:29.759737    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-963936-29895765-52lrg"
Nov 03 22:45:29.759973 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:29.759751    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-963936-29895765-52lrg" event={"ID":"57ada6e6-75cf-47a4-87d6-8ad44c0e442e","Type":"ContainerDied","Data":"d2035bb4c1a686df1b9dd0ea70e7b75c8116d4525c8f94773248fb08a78e4887"}
Nov 03 22:45:29.759973 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:29.759937    4431 scope.go:122] "RemoveContainer" containerID="52434fb2cd5fcc0a93ac07c6b06f0ab1f886c53e5140c594ce7663f34ae08030"
Nov 03 22:45:29.765183 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:29.765153    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2d6076-29895765-5njwc" event={"ID":"8b5f2047-8624-42fb-9881-6d81c25867ee","Type":"ContainerDied","Data":"4de9da6ef477cf12e4453f052b9db674689f8439a2e56f81ae12a9f7be3f92e3"}
Nov 03 22:45:29.765399 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:29.765175    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2d6076-29895765-5njwc"
Nov 03 22:45:29.771930 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:29.771904    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7940ea-29895765-nvrxh"
Nov 03 22:45:29.772044 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:29.771928    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7940ea-29895765-nvrxh" event={"ID":"4b6f92d6-3fbb-48fe-854f-6b5b899f56d1","Type":"ContainerDied","Data":"e55d69ec0e5bf96f3a6cd81c16b344e19075f4c4950941e1fa9d7e414911c8f4"}
Nov 03 22:45:29.774266 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:29.774248    4431 scope.go:122] "RemoveContainer" containerID="25a91d81884ac6da31c07b09c71204e96c06e917d5910f71c8ca64b422a1eaca"
Nov 03 22:45:29.786035 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:29.786013    4431 scope.go:122] "RemoveContainer" containerID="ddb759ca587608a9d63a252de252ce3cf89790f8565a96cafc3dada30dc1e2ca"
Nov 03 22:45:29.812773 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:29.812744    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:45:29.814436 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:29.814412    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-963936-29895765-52lrg"]
Nov 03 22:45:29.816355 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:45:29.816329    4431 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57ada6e6_75cf_47a4_87d6_8ad44c0e442e.slice\": RecentStats: unable to find data in memory cache]"
Nov 03 22:45:29.862038 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:29.861963    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-963936-29895765-52lrg"]
Nov 03 22:45:29.915843 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:29.915801    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7940ea-29895765-nvrxh"]
Nov 03 22:45:29.920642 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:29.920607    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7940ea-29895765-nvrxh"]
Nov 03 22:45:29.950723 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:29.950686    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2d6076-29895765-5njwc"]
Nov 03 22:45:29.954695 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:29.954667    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2d6076-29895765-5njwc"]
Nov 03 22:45:30.090045 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:30.089972    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d201ce0-fde0-494b-988a-b19d31dbbcda" path="/var/lib/kubelet/pods/0d201ce0-fde0-494b-988a-b19d31dbbcda/volumes"
Nov 03 22:45:30.090544 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:30.090526    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b6f92d6-3fbb-48fe-854f-6b5b899f56d1" path="/var/lib/kubelet/pods/4b6f92d6-3fbb-48fe-854f-6b5b899f56d1/volumes"
Nov 03 22:45:30.090948 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:30.090934    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57ada6e6-75cf-47a4-87d6-8ad44c0e442e" path="/var/lib/kubelet/pods/57ada6e6-75cf-47a4-87d6-8ad44c0e442e/volumes"
Nov 03 22:45:30.091769 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:30.091751    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ef57864-5184-4b21-8fae-201006734abf" path="/var/lib/kubelet/pods/7ef57864-5184-4b21-8fae-201006734abf/volumes"
Nov 03 22:45:30.092253 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:30.092230    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b5f2047-8624-42fb-9881-6d81c25867ee" path="/var/lib/kubelet/pods/8b5f2047-8624-42fb-9881-6d81c25867ee/volumes"
Nov 03 22:45:31.261201 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:45:31.261165    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:45:31.549950 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:31.549853    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:45:33.157238 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:33.144311    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:45:33.841376 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:33.841340    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:45:33.986100 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:33.986060    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:45:34.546610 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:34.546566    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:45:34.718400 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:34.718361    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:45:36.347316 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:36.347278    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:45:37.938953 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:37.938913    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:45:38.117943 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:38.117902    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:45:38.974011 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:38.973972    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:45:39.600116 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:39.600075    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:45:40.332892 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:40.332856    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:45:40.477930 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:40.477899    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:45:41.324913 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:41.324838    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:45:42.922755 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:42.922722    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:45:44.494942 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:44.494897    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:45:46.032089 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:46.032056    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:45:47.705891 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:47.705858    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:45:48.119695 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:48.119603    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:45:49.385487 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:49.385415    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:45:51.315968 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:51.315934    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:45:53.009172 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:53.009136    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:45:53.230305 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:53.230252    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:45:54.658731 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:54.658686    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:45:56.224493 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:56.224456    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:45:57.762024 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:57.761975    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:45:59.267131 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:45:59.265991    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:46:00.718033 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:00.717999    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:46:01.261249 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:46:01.261187    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:46:02.219389 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:02.219355    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:46:03.812483 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:03.812448    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:46:05.434146 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:05.434093    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:46:07.012623 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:07.012580    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:46:08.622892 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:08.622848    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:46:10.173633 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:10.173599    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:46:11.918191 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:11.918154    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:46:13.605078 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:13.605039    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:46:15.169763 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:15.169731    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:46:16.719153 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:16.719092    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:46:18.389170 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:18.389129    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:46:19.846896 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:19.846861    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:46:21.149866 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:21.149831    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:46:21.699157 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:21.699119    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:46:21.855809 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:21.855758    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:46:22.006087 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:22.006007    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:46:22.494205 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:22.494165    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:46:23.455930 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:23.455894    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:46:23.601820 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:23.601771    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:46:24.493006 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:24.492973    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:46:24.631317 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:24.631286    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:46:25.101837 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:25.101802    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:46:26.579568 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:26.579516    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:46:28.491751 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:28.491708    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:46:29.948766 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:29.948736    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:46:30.094407 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:30.094376    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:46:30.267812 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:30.267697    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss"]
Nov 03 22:46:30.268119 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:30.268053    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss" podUID="10b331d4-81e4-4243-a645-69d95958db00" containerName="istio-proxy" containerID="cri-o://01351340e51ce0fb93b3c787cb2f6052ae23c02b50a12649390f2aaee60d4b4f" gracePeriod=30
Nov 03 22:46:30.314128 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:30.314086    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:46:30.463777 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:30.463741    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:46:31.261320 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:46:31.261279    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:46:32.274105 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:32.274069    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:46:33.828713 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:33.828666    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:46:35.411919 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:35.411883    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:46:35.597549 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:35.597519    4431 generic.go:356] "Generic (PLEG): container finished" podID="10b331d4-81e4-4243-a645-69d95958db00" containerID="01351340e51ce0fb93b3c787cb2f6052ae23c02b50a12649390f2aaee60d4b4f" exitCode=0
Nov 03 22:46:35.597729 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:35.597597    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss" event={"ID":"10b331d4-81e4-4243-a645-69d95958db00","Type":"ContainerDied","Data":"01351340e51ce0fb93b3c787cb2f6052ae23c02b50a12649390f2aaee60d4b4f"}
Nov 03 22:46:35.597729 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:35.597664    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss" event={"ID":"10b331d4-81e4-4243-a645-69d95958db00","Type":"ContainerDied","Data":"9a0852a4a0ebd7cc9fb66d15521a3c16abd863fd23e74dd1f873e32001fe2925"}
Nov 03 22:46:35.597729 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:35.597691    4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a0852a4a0ebd7cc9fb66d15521a3c16abd863fd23e74dd1f873e32001fe2925"
Nov 03 22:46:35.597729 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:35.597708    4431 scope.go:122] "RemoveContainer" containerID="9f24b059f48447fac205ac83277f336c7be557b4493c31260420aa82e575e556"
Nov 03 22:46:35.599400 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:35.599379    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss"
Nov 03 22:46:35.645175 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:35.644845    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10b331d4-81e4-4243-a645-69d95958db00-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b331d4-81e4-4243-a645-69d95958db00-credential-socket\") pod \"10b331d4-81e4-4243-a645-69d95958db00\" (UID: \"10b331d4-81e4-4243-a645-69d95958db00\") "
Nov 03 22:46:35.645175 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:35.644941    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10b331d4-81e4-4243-a645-69d95958db00-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10b331d4-81e4-4243-a645-69d95958db00-workload-certs\") pod \"10b331d4-81e4-4243-a645-69d95958db00\" (UID: \"10b331d4-81e4-4243-a645-69d95958db00\") "
Nov 03 22:46:35.645175 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:35.644998    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10b331d4-81e4-4243-a645-69d95958db00-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10b331d4-81e4-4243-a645-69d95958db00-istio-envoy\") pod \"10b331d4-81e4-4243-a645-69d95958db00\" (UID: \"10b331d4-81e4-4243-a645-69d95958db00\") "
Nov 03 22:46:35.645175 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:35.645083    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/10b331d4-81e4-4243-a645-69d95958db00-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10b331d4-81e4-4243-a645-69d95958db00-istio-podinfo\") pod \"10b331d4-81e4-4243-a645-69d95958db00\" (UID: \"10b331d4-81e4-4243-a645-69d95958db00\") "
Nov 03 22:46:35.645175 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:35.645112    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10b331d4-81e4-4243-a645-69d95958db00-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10b331d4-81e4-4243-a645-69d95958db00-istiod-ca-cert\") pod \"10b331d4-81e4-4243-a645-69d95958db00\" (UID: \"10b331d4-81e4-4243-a645-69d95958db00\") "
Nov 03 22:46:35.645548 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:35.645246    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10b331d4-81e4-4243-a645-69d95958db00-kube-api-access-lnsrm\" (UniqueName: \"kubernetes.io/projected/10b331d4-81e4-4243-a645-69d95958db00-kube-api-access-lnsrm\") pod \"10b331d4-81e4-4243-a645-69d95958db00\" (UID: \"10b331d4-81e4-4243-a645-69d95958db00\") "
Nov 03 22:46:35.645548 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:35.645352    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10b331d4-81e4-4243-a645-69d95958db00-istio-token\" (UniqueName: \"kubernetes.io/projected/10b331d4-81e4-4243-a645-69d95958db00-istio-token\") pod \"10b331d4-81e4-4243-a645-69d95958db00\" (UID: \"10b331d4-81e4-4243-a645-69d95958db00\") "
Nov 03 22:46:35.645548 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:35.645499    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10b331d4-81e4-4243-a645-69d95958db00-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10b331d4-81e4-4243-a645-69d95958db00-istio-data\") pod \"10b331d4-81e4-4243-a645-69d95958db00\" (UID: \"10b331d4-81e4-4243-a645-69d95958db00\") "
Nov 03 22:46:35.645697 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:35.645567    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10b331d4-81e4-4243-a645-69d95958db00-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b331d4-81e4-4243-a645-69d95958db00-workload-socket\") pod \"10b331d4-81e4-4243-a645-69d95958db00\" (UID: \"10b331d4-81e4-4243-a645-69d95958db00\") "
Nov 03 22:46:35.650238 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:35.647237    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10b331d4-81e4-4243-a645-69d95958db00-workload-socket" pod "10b331d4-81e4-4243-a645-69d95958db00" (UID: "10b331d4-81e4-4243-a645-69d95958db00"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:46:35.650238 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:35.647457    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10b331d4-81e4-4243-a645-69d95958db00-credential-socket" pod "10b331d4-81e4-4243-a645-69d95958db00" (UID: "10b331d4-81e4-4243-a645-69d95958db00"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:46:35.650238 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:35.647625    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10b331d4-81e4-4243-a645-69d95958db00-workload-certs" pod "10b331d4-81e4-4243-a645-69d95958db00" (UID: "10b331d4-81e4-4243-a645-69d95958db00"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:46:35.650238 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:35.650159    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10b331d4-81e4-4243-a645-69d95958db00-istio-envoy" pod "10b331d4-81e4-4243-a645-69d95958db00" (UID: "10b331d4-81e4-4243-a645-69d95958db00"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:46:35.652590 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:35.652562    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/10b331d4-81e4-4243-a645-69d95958db00-istio-podinfo" pod "10b331d4-81e4-4243-a645-69d95958db00" (UID: "10b331d4-81e4-4243-a645-69d95958db00"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 22:46:35.652908 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:35.652874    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10b331d4-81e4-4243-a645-69d95958db00-istiod-ca-cert" pod "10b331d4-81e4-4243-a645-69d95958db00" (UID: "10b331d4-81e4-4243-a645-69d95958db00"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:46:35.655433 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:35.655390    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10b331d4-81e4-4243-a645-69d95958db00-kube-api-access-lnsrm" pod "10b331d4-81e4-4243-a645-69d95958db00" (UID: "10b331d4-81e4-4243-a645-69d95958db00"). InnerVolumeSpecName "kube-api-access-lnsrm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:46:35.657318 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:35.657292    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10b331d4-81e4-4243-a645-69d95958db00-istio-data" pod "10b331d4-81e4-4243-a645-69d95958db00" (UID: "10b331d4-81e4-4243-a645-69d95958db00"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:46:35.658080 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:35.658045    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10b331d4-81e4-4243-a645-69d95958db00-istio-token" pod "10b331d4-81e4-4243-a645-69d95958db00" (UID: "10b331d4-81e4-4243-a645-69d95958db00"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:46:35.748870 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:35.748834    4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b331d4-81e4-4243-a645-69d95958db00-credential-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:46:35.748870 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:35.748867    4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10b331d4-81e4-4243-a645-69d95958db00-workload-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:46:35.748870 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:35.748877    4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10b331d4-81e4-4243-a645-69d95958db00-istio-envoy\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:46:35.749124 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:35.748885    4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10b331d4-81e4-4243-a645-69d95958db00-istio-podinfo\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:46:35.749124 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:35.748895    4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10b331d4-81e4-4243-a645-69d95958db00-istiod-ca-cert\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:46:35.749124 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:35.748903    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnsrm\" (UniqueName: \"kubernetes.io/projected/10b331d4-81e4-4243-a645-69d95958db00-kube-api-access-lnsrm\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:46:35.749124 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:35.748911    4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10b331d4-81e4-4243-a645-69d95958db00-istio-token\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:46:35.749124 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:35.748919    4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10b331d4-81e4-4243-a645-69d95958db00-istio-data\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:46:35.749124 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:35.748926    4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b331d4-81e4-4243-a645-69d95958db00-workload-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:46:36.606587 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:36.606554    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss"
Nov 03 22:46:36.703893 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:36.703854    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss"]
Nov 03 22:46:36.707983 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:36.707959    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-4b5e2b-openshift-default-7496d87895-f5vss"]
Nov 03 22:46:36.899419 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:36.899329    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:46:38.091718 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:38.091682    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10b331d4-81e4-4243-a645-69d95958db00" path="/var/lib/kubelet/pods/10b331d4-81e4-4243-a645-69d95958db00/volumes"
Nov 03 22:46:38.423988 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:38.423954    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:46:40.003016 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:40.002975    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:46:41.279402 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:41.279361    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2d6076-d576997dd-rh48p"]
Nov 03 22:46:41.464656 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:41.464623    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:46:41.612633 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:41.612548    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:46:42.030248 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:42.024322    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84"]
Nov 03 22:46:42.030248 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:42.024634    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84" podUID="8af62636-7952-421c-8302-0af2cfefe3b7" containerName="istio-proxy" containerID="cri-o://220a438bc6516a2c859e660e15fc26b7846372be746b740b240b71297a6ae1ff" gracePeriod=30
Nov 03 22:46:43.224021 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:43.223967    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:46:43.438080 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:43.438046    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz"]
Nov 03 22:46:43.438313 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:43.438284    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz" podUID="892720cc-753b-43d6-ad3b-0494dff25623" containerName="istio-proxy" containerID="cri-o://e863a9a320c2c2afdc4ebbb5f294c33044671e9cc1341972243229c285e3190b" gracePeriod=30
Nov 03 22:46:44.809366 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:44.809325    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:46:45.865198 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:45.865158    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6"]
Nov 03 22:46:45.865718 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:45.865409    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6" podUID="7dde620e-e87f-4f87-984e-6358a642a100" containerName="istio-proxy" containerID="cri-o://75d3ccc913873d606bba13bb7360634f057d48dfdd96551ebf4e2803cc8981a2" gracePeriod=30
Nov 03 22:46:46.280015 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:46.279972    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-2d6076-d576997dd-rh48p" podUID="9749991e-8f89-488c-9f71-e7da0c122683" containerName="payload-processing" containerID="cri-o://d74ce5a66b2142027aab3d1b95727352ee5af095ad4c4cca92e9db6559d670e9" gracePeriod=25
Nov 03 22:46:46.389390 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:46.389355    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:46:46.712566 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:46.712528    4431 patch_prober.go:29] interesting pod/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 22:46:46.712742 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:46.712591    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84" podUID="8af62636-7952-421c-8302-0af2cfefe3b7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 22:46:47.690869 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:47.690834    4431 generic.go:356] "Generic (PLEG): container finished" podID="8af62636-7952-421c-8302-0af2cfefe3b7" containerID="220a438bc6516a2c859e660e15fc26b7846372be746b740b240b71297a6ae1ff" exitCode=0
Nov 03 22:46:47.691282 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:47.691000    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84" event={"ID":"8af62636-7952-421c-8302-0af2cfefe3b7","Type":"ContainerDied","Data":"220a438bc6516a2c859e660e15fc26b7846372be746b740b240b71297a6ae1ff"}
Nov 03 22:46:47.691282 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:47.691052    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84" event={"ID":"8af62636-7952-421c-8302-0af2cfefe3b7","Type":"ContainerDied","Data":"31a8df8d60cc16e761fc7da94548d2b9d9d1908dc91aff6ceae490963e7691ed"}
Nov 03 22:46:47.691282 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:47.691075    4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31a8df8d60cc16e761fc7da94548d2b9d9d1908dc91aff6ceae490963e7691ed"
Nov 03 22:46:47.713978 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:47.713952    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84"
Nov 03 22:46:47.843736 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:47.843637    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8af62636-7952-421c-8302-0af2cfefe3b7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8af62636-7952-421c-8302-0af2cfefe3b7-istiod-ca-cert\") pod \"8af62636-7952-421c-8302-0af2cfefe3b7\" (UID: \"8af62636-7952-421c-8302-0af2cfefe3b7\") "
Nov 03 22:46:47.843736 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:47.843690    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8af62636-7952-421c-8302-0af2cfefe3b7-kube-api-access-7r9sm\" (UniqueName: \"kubernetes.io/projected/8af62636-7952-421c-8302-0af2cfefe3b7-kube-api-access-7r9sm\") pod \"8af62636-7952-421c-8302-0af2cfefe3b7\" (UID: \"8af62636-7952-421c-8302-0af2cfefe3b7\") "
Nov 03 22:46:47.843979 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:47.843752    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8af62636-7952-421c-8302-0af2cfefe3b7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8af62636-7952-421c-8302-0af2cfefe3b7-workload-socket\") pod \"8af62636-7952-421c-8302-0af2cfefe3b7\" (UID: \"8af62636-7952-421c-8302-0af2cfefe3b7\") "
Nov 03 22:46:47.843979 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:47.843947    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8af62636-7952-421c-8302-0af2cfefe3b7-istio-token\" (UniqueName: \"kubernetes.io/projected/8af62636-7952-421c-8302-0af2cfefe3b7-istio-token\") pod \"8af62636-7952-421c-8302-0af2cfefe3b7\" (UID: \"8af62636-7952-421c-8302-0af2cfefe3b7\") "
Nov 03 22:46:47.844119 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:47.843989    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8af62636-7952-421c-8302-0af2cfefe3b7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8af62636-7952-421c-8302-0af2cfefe3b7-istio-data\") pod \"8af62636-7952-421c-8302-0af2cfefe3b7\" (UID: \"8af62636-7952-421c-8302-0af2cfefe3b7\") "
Nov 03 22:46:47.844119 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:47.844029    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8af62636-7952-421c-8302-0af2cfefe3b7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8af62636-7952-421c-8302-0af2cfefe3b7-istio-podinfo\") pod \"8af62636-7952-421c-8302-0af2cfefe3b7\" (UID: \"8af62636-7952-421c-8302-0af2cfefe3b7\") "
Nov 03 22:46:47.844246 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:47.844166    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8af62636-7952-421c-8302-0af2cfefe3b7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8af62636-7952-421c-8302-0af2cfefe3b7-istio-envoy\") pod \"8af62636-7952-421c-8302-0af2cfefe3b7\" (UID: \"8af62636-7952-421c-8302-0af2cfefe3b7\") "
Nov 03 22:46:47.844246 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:47.844176    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8af62636-7952-421c-8302-0af2cfefe3b7-workload-socket" pod "8af62636-7952-421c-8302-0af2cfefe3b7" (UID: "8af62636-7952-421c-8302-0af2cfefe3b7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:46:47.844246 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:47.844209    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8af62636-7952-421c-8302-0af2cfefe3b7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8af62636-7952-421c-8302-0af2cfefe3b7-workload-certs\") pod \"8af62636-7952-421c-8302-0af2cfefe3b7\" (UID: \"8af62636-7952-421c-8302-0af2cfefe3b7\") "
Nov 03 22:46:47.844422 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:47.844298    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8af62636-7952-421c-8302-0af2cfefe3b7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8af62636-7952-421c-8302-0af2cfefe3b7-credential-socket\") pod \"8af62636-7952-421c-8302-0af2cfefe3b7\" (UID: \"8af62636-7952-421c-8302-0af2cfefe3b7\") "
Nov 03 22:46:47.844482 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:47.844456    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8af62636-7952-421c-8302-0af2cfefe3b7-istio-data" pod "8af62636-7952-421c-8302-0af2cfefe3b7" (UID: "8af62636-7952-421c-8302-0af2cfefe3b7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:46:47.844546 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:47.844528    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8af62636-7952-421c-8302-0af2cfefe3b7-istiod-ca-cert" pod "8af62636-7952-421c-8302-0af2cfefe3b7" (UID: "8af62636-7952-421c-8302-0af2cfefe3b7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:46:47.845029 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:47.844972    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8af62636-7952-421c-8302-0af2cfefe3b7-workload-certs" pod "8af62636-7952-421c-8302-0af2cfefe3b7" (UID: "8af62636-7952-421c-8302-0af2cfefe3b7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:46:47.845145 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:47.845044    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8af62636-7952-421c-8302-0af2cfefe3b7-credential-socket" pod "8af62636-7952-421c-8302-0af2cfefe3b7" (UID: "8af62636-7952-421c-8302-0af2cfefe3b7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:46:47.845246 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:47.845164    4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8af62636-7952-421c-8302-0af2cfefe3b7-workload-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:46:47.845246 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:47.845185    4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8af62636-7952-421c-8302-0af2cfefe3b7-credential-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:46:47.845246 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:47.845200    4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8af62636-7952-421c-8302-0af2cfefe3b7-istiod-ca-cert\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:46:47.845246 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:47.845227    4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8af62636-7952-421c-8302-0af2cfefe3b7-workload-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:46:47.845246 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:47.845239    4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8af62636-7952-421c-8302-0af2cfefe3b7-istio-data\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:46:47.847314 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:47.847291    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8af62636-7952-421c-8302-0af2cfefe3b7-istio-podinfo" pod "8af62636-7952-421c-8302-0af2cfefe3b7" (UID: "8af62636-7952-421c-8302-0af2cfefe3b7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 22:46:47.847426 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:47.847353    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8af62636-7952-421c-8302-0af2cfefe3b7-istio-envoy" pod "8af62636-7952-421c-8302-0af2cfefe3b7" (UID: "8af62636-7952-421c-8302-0af2cfefe3b7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:46:47.847785 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:47.847756    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8af62636-7952-421c-8302-0af2cfefe3b7-istio-token" pod "8af62636-7952-421c-8302-0af2cfefe3b7" (UID: "8af62636-7952-421c-8302-0af2cfefe3b7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:46:47.856606 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:47.856585    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8af62636-7952-421c-8302-0af2cfefe3b7-kube-api-access-7r9sm" pod "8af62636-7952-421c-8302-0af2cfefe3b7" (UID: "8af62636-7952-421c-8302-0af2cfefe3b7"). InnerVolumeSpecName "kube-api-access-7r9sm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:46:47.876871 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:47.876843    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:46:47.947454 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:47.947338    4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8af62636-7952-421c-8302-0af2cfefe3b7-istio-envoy\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:46:47.947454 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:47.947385    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7r9sm\" (UniqueName: \"kubernetes.io/projected/8af62636-7952-421c-8302-0af2cfefe3b7-kube-api-access-7r9sm\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:46:47.947454 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:47.947401    4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8af62636-7952-421c-8302-0af2cfefe3b7-istio-token\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:46:47.947454 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:47.947415    4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8af62636-7952-421c-8302-0af2cfefe3b7-istio-podinfo\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:46:47.951949 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:47.951647    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-2d6076-d576997dd-rh48p" podUID="9749991e-8f89-488c-9f71-e7da0c122683" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 22:46:48.244293 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:48.244201    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:46:48.701316 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:48.701196    4431 generic.go:356] "Generic (PLEG): container finished" podID="892720cc-753b-43d6-ad3b-0494dff25623" containerID="e863a9a320c2c2afdc4ebbb5f294c33044671e9cc1341972243229c285e3190b" exitCode=0
Nov 03 22:46:48.703446 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:48.701505    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84"
Nov 03 22:46:48.703446 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:48.701756    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz" event={"ID":"892720cc-753b-43d6-ad3b-0494dff25623","Type":"ContainerDied","Data":"e863a9a320c2c2afdc4ebbb5f294c33044671e9cc1341972243229c285e3190b"}
Nov 03 22:46:48.703446 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:48.701820    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz" event={"ID":"892720cc-753b-43d6-ad3b-0494dff25623","Type":"ContainerDied","Data":"b8cc2f784af8f2c518c5d4e8648a01dd71e6503f84863a6bb52ffd6933e5701b"}
Nov 03 22:46:48.703446 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:48.701846    4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8cc2f784af8f2c518c5d4e8648a01dd71e6503f84863a6bb52ffd6933e5701b"
Nov 03 22:46:48.728986 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:48.728961    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz"
Nov 03 22:46:48.763829 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:48.763800    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84"]
Nov 03 22:46:48.775644 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:48.775613    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-8xg84"]
Nov 03 22:46:48.861045 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:48.861010    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/892720cc-753b-43d6-ad3b-0494dff25623-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/892720cc-753b-43d6-ad3b-0494dff25623-credential-socket\") pod \"892720cc-753b-43d6-ad3b-0494dff25623\" (UID: \"892720cc-753b-43d6-ad3b-0494dff25623\") "
Nov 03 22:46:48.861256 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:48.861065    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/892720cc-753b-43d6-ad3b-0494dff25623-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/892720cc-753b-43d6-ad3b-0494dff25623-istio-podinfo\") pod \"892720cc-753b-43d6-ad3b-0494dff25623\" (UID: \"892720cc-753b-43d6-ad3b-0494dff25623\") "
Nov 03 22:46:48.861256 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:48.861112    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/892720cc-753b-43d6-ad3b-0494dff25623-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/892720cc-753b-43d6-ad3b-0494dff25623-workload-certs\") pod \"892720cc-753b-43d6-ad3b-0494dff25623\" (UID: \"892720cc-753b-43d6-ad3b-0494dff25623\") "
Nov 03 22:46:48.861256 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:48.861180    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/892720cc-753b-43d6-ad3b-0494dff25623-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/892720cc-753b-43d6-ad3b-0494dff25623-workload-socket\") pod \"892720cc-753b-43d6-ad3b-0494dff25623\" (UID: \"892720cc-753b-43d6-ad3b-0494dff25623\") "
Nov 03 22:46:48.861464 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:48.861388    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/892720cc-753b-43d6-ad3b-0494dff25623-credential-socket" pod "892720cc-753b-43d6-ad3b-0494dff25623" (UID: "892720cc-753b-43d6-ad3b-0494dff25623"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:46:48.861632 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:48.861559    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/892720cc-753b-43d6-ad3b-0494dff25623-workload-certs" pod "892720cc-753b-43d6-ad3b-0494dff25623" (UID: "892720cc-753b-43d6-ad3b-0494dff25623"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:46:48.861632 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:48.861619    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/892720cc-753b-43d6-ad3b-0494dff25623-workload-socket" pod "892720cc-753b-43d6-ad3b-0494dff25623" (UID: "892720cc-753b-43d6-ad3b-0494dff25623"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:46:48.861769 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:48.861368    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/892720cc-753b-43d6-ad3b-0494dff25623-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/892720cc-753b-43d6-ad3b-0494dff25623-istio-data\") pod \"892720cc-753b-43d6-ad3b-0494dff25623\" (UID: \"892720cc-753b-43d6-ad3b-0494dff25623\") "
Nov 03 22:46:48.861851 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:48.861830    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/892720cc-753b-43d6-ad3b-0494dff25623-istio-data" pod "892720cc-753b-43d6-ad3b-0494dff25623" (UID: "892720cc-753b-43d6-ad3b-0494dff25623"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:46:48.861939 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:48.861917    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/892720cc-753b-43d6-ad3b-0494dff25623-kube-api-access-vrq5p\" (UniqueName: \"kubernetes.io/projected/892720cc-753b-43d6-ad3b-0494dff25623-kube-api-access-vrq5p\") pod \"892720cc-753b-43d6-ad3b-0494dff25623\" (UID: \"892720cc-753b-43d6-ad3b-0494dff25623\") "
Nov 03 22:46:48.861999 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:48.861976    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/892720cc-753b-43d6-ad3b-0494dff25623-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/892720cc-753b-43d6-ad3b-0494dff25623-istio-envoy\") pod \"892720cc-753b-43d6-ad3b-0494dff25623\" (UID: \"892720cc-753b-43d6-ad3b-0494dff25623\") "
Nov 03 22:46:48.862056 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:48.862012    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/892720cc-753b-43d6-ad3b-0494dff25623-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/892720cc-753b-43d6-ad3b-0494dff25623-istiod-ca-cert\") pod \"892720cc-753b-43d6-ad3b-0494dff25623\" (UID: \"892720cc-753b-43d6-ad3b-0494dff25623\") "
Nov 03 22:46:48.862138 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:48.862121    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/892720cc-753b-43d6-ad3b-0494dff25623-istio-token\" (UniqueName: \"kubernetes.io/projected/892720cc-753b-43d6-ad3b-0494dff25623-istio-token\") pod \"892720cc-753b-43d6-ad3b-0494dff25623\" (UID: \"892720cc-753b-43d6-ad3b-0494dff25623\") "
Nov 03 22:46:48.862572 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:48.862543    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/892720cc-753b-43d6-ad3b-0494dff25623-istiod-ca-cert" pod "892720cc-753b-43d6-ad3b-0494dff25623" (UID: "892720cc-753b-43d6-ad3b-0494dff25623"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:46:48.862933 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:48.862908    4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/892720cc-753b-43d6-ad3b-0494dff25623-istiod-ca-cert\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:46:48.863089 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:48.863074    4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/892720cc-753b-43d6-ad3b-0494dff25623-credential-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:46:48.863204 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:48.863190    4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/892720cc-753b-43d6-ad3b-0494dff25623-workload-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:46:48.863340 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:48.863326    4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/892720cc-753b-43d6-ad3b-0494dff25623-workload-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:46:48.863462 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:48.863449    4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/892720cc-753b-43d6-ad3b-0494dff25623-istio-data\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:46:48.863824 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:48.863800    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/892720cc-753b-43d6-ad3b-0494dff25623-istio-podinfo" pod "892720cc-753b-43d6-ad3b-0494dff25623" (UID: "892720cc-753b-43d6-ad3b-0494dff25623"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 22:46:48.864707 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:48.864682    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/892720cc-753b-43d6-ad3b-0494dff25623-istio-token" pod "892720cc-753b-43d6-ad3b-0494dff25623" (UID: "892720cc-753b-43d6-ad3b-0494dff25623"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:46:48.865675 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:48.865649    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/892720cc-753b-43d6-ad3b-0494dff25623-istio-envoy" pod "892720cc-753b-43d6-ad3b-0494dff25623" (UID: "892720cc-753b-43d6-ad3b-0494dff25623"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:46:48.866333 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:48.866309    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/892720cc-753b-43d6-ad3b-0494dff25623-kube-api-access-vrq5p" pod "892720cc-753b-43d6-ad3b-0494dff25623" (UID: "892720cc-753b-43d6-ad3b-0494dff25623"). InnerVolumeSpecName "kube-api-access-vrq5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:46:48.965822 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:48.965740    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrq5p\" (UniqueName: \"kubernetes.io/projected/892720cc-753b-43d6-ad3b-0494dff25623-kube-api-access-vrq5p\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:46:48.965822 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:48.965771    4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/892720cc-753b-43d6-ad3b-0494dff25623-istio-envoy\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:46:48.965822 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:48.965780    4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/892720cc-753b-43d6-ad3b-0494dff25623-istio-token\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:46:48.965822 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:48.965788    4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/892720cc-753b-43d6-ad3b-0494dff25623-istio-podinfo\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:46:49.511765 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:49.511725    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:46:49.708845 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:49.708813    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz"
Nov 03 22:46:49.764169 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:49.764092    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz"]
Nov 03 22:46:49.774379 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:49.774349    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-fd2cz"]
Nov 03 22:46:50.092558 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:50.092470    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="892720cc-753b-43d6-ad3b-0494dff25623" path="/var/lib/kubelet/pods/892720cc-753b-43d6-ad3b-0494dff25623/volumes"
Nov 03 22:46:50.093148 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:50.093113    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8af62636-7952-421c-8302-0af2cfefe3b7" path="/var/lib/kubelet/pods/8af62636-7952-421c-8302-0af2cfefe3b7/volumes"
Nov 03 22:46:51.043872 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:51.043789    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:46:51.220932 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:51.220908    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6"
Nov 03 22:46:51.301532 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:51.301496    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7dde620e-e87f-4f87-984e-6358a642a100-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dde620e-e87f-4f87-984e-6358a642a100-istiod-ca-cert\") pod \"7dde620e-e87f-4f87-984e-6358a642a100\" (UID: \"7dde620e-e87f-4f87-984e-6358a642a100\") "
Nov 03 22:46:51.301738 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:51.301557    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dde620e-e87f-4f87-984e-6358a642a100-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dde620e-e87f-4f87-984e-6358a642a100-istio-envoy\") pod \"7dde620e-e87f-4f87-984e-6358a642a100\" (UID: \"7dde620e-e87f-4f87-984e-6358a642a100\") "
Nov 03 22:46:51.301738 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:51.301628    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dde620e-e87f-4f87-984e-6358a642a100-kube-api-access-wdt8m\" (UniqueName: \"kubernetes.io/projected/7dde620e-e87f-4f87-984e-6358a642a100-kube-api-access-wdt8m\") pod \"7dde620e-e87f-4f87-984e-6358a642a100\" (UID: \"7dde620e-e87f-4f87-984e-6358a642a100\") "
Nov 03 22:46:51.301738 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:51.301675    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dde620e-e87f-4f87-984e-6358a642a100-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dde620e-e87f-4f87-984e-6358a642a100-workload-socket\") pod \"7dde620e-e87f-4f87-984e-6358a642a100\" (UID: \"7dde620e-e87f-4f87-984e-6358a642a100\") "
Nov 03 22:46:51.301738 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:51.301703    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dde620e-e87f-4f87-984e-6358a642a100-istio-token\" (UniqueName: \"kubernetes.io/projected/7dde620e-e87f-4f87-984e-6358a642a100-istio-token\") pod \"7dde620e-e87f-4f87-984e-6358a642a100\" (UID: \"7dde620e-e87f-4f87-984e-6358a642a100\") "
Nov 03 22:46:51.301963 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:51.301778    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dde620e-e87f-4f87-984e-6358a642a100-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dde620e-e87f-4f87-984e-6358a642a100-credential-socket\") pod \"7dde620e-e87f-4f87-984e-6358a642a100\" (UID: \"7dde620e-e87f-4f87-984e-6358a642a100\") "
Nov 03 22:46:51.301963 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:51.301834    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dde620e-e87f-4f87-984e-6358a642a100-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dde620e-e87f-4f87-984e-6358a642a100-istio-data\") pod \"7dde620e-e87f-4f87-984e-6358a642a100\" (UID: \"7dde620e-e87f-4f87-984e-6358a642a100\") "
Nov 03 22:46:51.301963 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:51.301895    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dde620e-e87f-4f87-984e-6358a642a100-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dde620e-e87f-4f87-984e-6358a642a100-workload-certs\") pod \"7dde620e-e87f-4f87-984e-6358a642a100\" (UID: \"7dde620e-e87f-4f87-984e-6358a642a100\") "
Nov 03 22:46:51.301963 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:51.301937    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dde620e-e87f-4f87-984e-6358a642a100-istiod-ca-cert" pod "7dde620e-e87f-4f87-984e-6358a642a100" (UID: "7dde620e-e87f-4f87-984e-6358a642a100"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:46:51.302168 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:51.302044    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7dde620e-e87f-4f87-984e-6358a642a100-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dde620e-e87f-4f87-984e-6358a642a100-istio-podinfo\") pod \"7dde620e-e87f-4f87-984e-6358a642a100\" (UID: \"7dde620e-e87f-4f87-984e-6358a642a100\") "
Nov 03 22:46:51.302264 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:51.302241    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dde620e-e87f-4f87-984e-6358a642a100-workload-certs" pod "7dde620e-e87f-4f87-984e-6358a642a100" (UID: "7dde620e-e87f-4f87-984e-6358a642a100"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:46:51.302336 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:51.302287    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dde620e-e87f-4f87-984e-6358a642a100-workload-socket" pod "7dde620e-e87f-4f87-984e-6358a642a100" (UID: "7dde620e-e87f-4f87-984e-6358a642a100"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:46:51.302828 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:51.302802    4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dde620e-e87f-4f87-984e-6358a642a100-istiod-ca-cert\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:46:51.302828 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:51.302829    4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dde620e-e87f-4f87-984e-6358a642a100-workload-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:46:51.303073 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:51.302844    4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dde620e-e87f-4f87-984e-6358a642a100-workload-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:46:51.303472 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:51.303444    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dde620e-e87f-4f87-984e-6358a642a100-istio-data" pod "7dde620e-e87f-4f87-984e-6358a642a100" (UID: "7dde620e-e87f-4f87-984e-6358a642a100"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:46:51.303617 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:51.303512    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dde620e-e87f-4f87-984e-6358a642a100-credential-socket" pod "7dde620e-e87f-4f87-984e-6358a642a100" (UID: "7dde620e-e87f-4f87-984e-6358a642a100"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:46:51.304264 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:51.304174    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dde620e-e87f-4f87-984e-6358a642a100-istio-envoy" pod "7dde620e-e87f-4f87-984e-6358a642a100" (UID: "7dde620e-e87f-4f87-984e-6358a642a100"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:46:51.305093 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:51.305066    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7dde620e-e87f-4f87-984e-6358a642a100-istio-podinfo" pod "7dde620e-e87f-4f87-984e-6358a642a100" (UID: "7dde620e-e87f-4f87-984e-6358a642a100"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 22:46:51.305327 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:51.305299    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dde620e-e87f-4f87-984e-6358a642a100-istio-token" pod "7dde620e-e87f-4f87-984e-6358a642a100" (UID: "7dde620e-e87f-4f87-984e-6358a642a100"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:46:51.305802 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:51.305781    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dde620e-e87f-4f87-984e-6358a642a100-kube-api-access-wdt8m" pod "7dde620e-e87f-4f87-984e-6358a642a100" (UID: "7dde620e-e87f-4f87-984e-6358a642a100"). InnerVolumeSpecName "kube-api-access-wdt8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:46:51.404001 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:51.403966    4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dde620e-e87f-4f87-984e-6358a642a100-credential-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:46:51.404001 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:51.403997    4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dde620e-e87f-4f87-984e-6358a642a100-istio-data\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:46:51.404001 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:51.404006    4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dde620e-e87f-4f87-984e-6358a642a100-istio-podinfo\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:46:51.404265 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:51.404014    4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dde620e-e87f-4f87-984e-6358a642a100-istio-envoy\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:46:51.404265 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:51.404022    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdt8m\" (UniqueName: \"kubernetes.io/projected/7dde620e-e87f-4f87-984e-6358a642a100-kube-api-access-wdt8m\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:46:51.404265 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:51.404031    4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dde620e-e87f-4f87-984e-6358a642a100-istio-token\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:46:51.724507 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:51.724473    4431 generic.go:356] "Generic (PLEG): container finished" podID="7dde620e-e87f-4f87-984e-6358a642a100" containerID="75d3ccc913873d606bba13bb7360634f057d48dfdd96551ebf4e2803cc8981a2" exitCode=0
Nov 03 22:46:51.724691 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:51.724525    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6"
Nov 03 22:46:51.724691 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:51.724553    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6" event={"ID":"7dde620e-e87f-4f87-984e-6358a642a100","Type":"ContainerDied","Data":"75d3ccc913873d606bba13bb7360634f057d48dfdd96551ebf4e2803cc8981a2"}
Nov 03 22:46:51.724691 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:51.724607    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6" event={"ID":"7dde620e-e87f-4f87-984e-6358a642a100","Type":"ContainerDied","Data":"40d49840b1c2132a88411cbedfe4c1648d0fc2b675edce31e27c40156e77b4be"}
Nov 03 22:46:51.724691 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:51.724629    4431 scope.go:122] "RemoveContainer" containerID="75d3ccc913873d606bba13bb7360634f057d48dfdd96551ebf4e2803cc8981a2"
Nov 03 22:46:51.736129 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:51.736110    4431 scope.go:122] "RemoveContainer" containerID="75d3ccc913873d606bba13bb7360634f057d48dfdd96551ebf4e2803cc8981a2"
Nov 03 22:46:51.736436 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:46:51.736418    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75d3ccc913873d606bba13bb7360634f057d48dfdd96551ebf4e2803cc8981a2\": container with ID starting with 75d3ccc913873d606bba13bb7360634f057d48dfdd96551ebf4e2803cc8981a2 not found: ID does not exist" containerID="75d3ccc913873d606bba13bb7360634f057d48dfdd96551ebf4e2803cc8981a2"
Nov 03 22:46:51.736490 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:51.736445    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75d3ccc913873d606bba13bb7360634f057d48dfdd96551ebf4e2803cc8981a2"} err="failed to get container status \"75d3ccc913873d606bba13bb7360634f057d48dfdd96551ebf4e2803cc8981a2\": rpc error: code = NotFound desc = could not find container \"75d3ccc913873d606bba13bb7360634f057d48dfdd96551ebf4e2803cc8981a2\": container with ID starting with 75d3ccc913873d606bba13bb7360634f057d48dfdd96551ebf4e2803cc8981a2 not found: ID does not exist"
Nov 03 22:46:51.790836 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:51.790808    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6"]
Nov 03 22:46:51.802374 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:51.802347    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-04988c-openshift-default-d99f984b9-zmsb6"]
Nov 03 22:46:52.090682 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:52.090606    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7dde620e-e87f-4f87-984e-6358a642a100" path="/var/lib/kubelet/pods/7dde620e-e87f-4f87-984e-6358a642a100/volumes"
Nov 03 22:46:52.524385 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:52.524341    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:46:53.355178 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:53.355140    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:46:54.202008 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:54.201968    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:46:55.716249 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:55.716155    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:46:57.387579 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:57.387545    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:46:57.932095 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:57.932057    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-2d6076-d576997dd-rh48p" podUID="9749991e-8f89-488c-9f71-e7da0c122683" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 22:46:58.509911 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:58.509878    4431 scope.go:122] "RemoveContainer" containerID="01351340e51ce0fb93b3c787cb2f6052ae23c02b50a12649390f2aaee60d4b4f"
Nov 03 22:46:58.520436 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:58.520413    4431 scope.go:122] "RemoveContainer" containerID="e863a9a320c2c2afdc4ebbb5f294c33044671e9cc1341972243229c285e3190b"
Nov 03 22:46:58.529917 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:58.529891    4431 scope.go:122] "RemoveContainer" containerID="220a438bc6516a2c859e660e15fc26b7846372be746b740b240b71297a6ae1ff"
Nov 03 22:46:58.927011 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:46:58.926980    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:47:00.402248 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:00.402197    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:47:01.261261 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:47:01.261225    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:47:01.869887 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:01.869857    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:47:03.449493 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:03.449453    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:47:04.031890 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:04.031858    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:47:04.615679 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:04.615641    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:47:05.248821 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:05.248780    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:47:06.594402 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:06.594372    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2d6076-d576997dd-rh48p"
Nov 03 22:47:06.666350 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:06.666321    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9749991e-8f89-488c-9f71-e7da0c122683-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9749991e-8f89-488c-9f71-e7da0c122683-plugins-config-volume\") pod \"9749991e-8f89-488c-9f71-e7da0c122683\" (UID: \"9749991e-8f89-488c-9f71-e7da0c122683\") "
Nov 03 22:47:06.666691 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:06.666671    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9749991e-8f89-488c-9f71-e7da0c122683-kube-api-access-4jqff\" (UniqueName: \"kubernetes.io/projected/9749991e-8f89-488c-9f71-e7da0c122683-kube-api-access-4jqff\") pod \"9749991e-8f89-488c-9f71-e7da0c122683\" (UID: \"9749991e-8f89-488c-9f71-e7da0c122683\") "
Nov 03 22:47:06.666807 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:06.666776    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9749991e-8f89-488c-9f71-e7da0c122683-plugins-config-volume" pod "9749991e-8f89-488c-9f71-e7da0c122683" (UID: "9749991e-8f89-488c-9f71-e7da0c122683"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:47:06.667353 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:06.667326    4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9749991e-8f89-488c-9f71-e7da0c122683-plugins-config-volume\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:47:06.668996 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:06.668962    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9749991e-8f89-488c-9f71-e7da0c122683-kube-api-access-4jqff" pod "9749991e-8f89-488c-9f71-e7da0c122683" (UID: "9749991e-8f89-488c-9f71-e7da0c122683"). InnerVolumeSpecName "kube-api-access-4jqff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:47:06.768744 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:06.768709    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jqff\" (UniqueName: \"kubernetes.io/projected/9749991e-8f89-488c-9f71-e7da0c122683-kube-api-access-4jqff\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:47:06.845513 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:06.845433    4431 generic.go:356] "Generic (PLEG): container finished" podID="9749991e-8f89-488c-9f71-e7da0c122683" containerID="d74ce5a66b2142027aab3d1b95727352ee5af095ad4c4cca92e9db6559d670e9" exitCode=0
Nov 03 22:47:06.845513 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:06.845484    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2d6076-d576997dd-rh48p"
Nov 03 22:47:06.845716 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:06.845520    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2d6076-d576997dd-rh48p" event={"ID":"9749991e-8f89-488c-9f71-e7da0c122683","Type":"ContainerDied","Data":"d74ce5a66b2142027aab3d1b95727352ee5af095ad4c4cca92e9db6559d670e9"}
Nov 03 22:47:06.845716 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:06.845595    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2d6076-d576997dd-rh48p" event={"ID":"9749991e-8f89-488c-9f71-e7da0c122683","Type":"ContainerDied","Data":"0df831cd754b8f01cc4ea5fd74f3d027624e305c0c6850dd130639a9f88bd86a"}
Nov 03 22:47:06.845716 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:06.845618    4431 scope.go:122] "RemoveContainer" containerID="d74ce5a66b2142027aab3d1b95727352ee5af095ad4c4cca92e9db6559d670e9"
Nov 03 22:47:06.858569 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:06.858544    4431 scope.go:122] "RemoveContainer" containerID="d74ce5a66b2142027aab3d1b95727352ee5af095ad4c4cca92e9db6559d670e9"
Nov 03 22:47:06.858881 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:47:06.858861    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d74ce5a66b2142027aab3d1b95727352ee5af095ad4c4cca92e9db6559d670e9\": container with ID starting with d74ce5a66b2142027aab3d1b95727352ee5af095ad4c4cca92e9db6559d670e9 not found: ID does not exist" containerID="d74ce5a66b2142027aab3d1b95727352ee5af095ad4c4cca92e9db6559d670e9"
Nov 03 22:47:06.858952 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:06.858887    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d74ce5a66b2142027aab3d1b95727352ee5af095ad4c4cca92e9db6559d670e9"} err="failed to get container status \"d74ce5a66b2142027aab3d1b95727352ee5af095ad4c4cca92e9db6559d670e9\": rpc error: code = NotFound desc = could not find container \"d74ce5a66b2142027aab3d1b95727352ee5af095ad4c4cca92e9db6559d670e9\": container with ID starting with d74ce5a66b2142027aab3d1b95727352ee5af095ad4c4cca92e9db6559d670e9 not found: ID does not exist"
Nov 03 22:47:06.898406 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:06.898372    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2d6076-d576997dd-rh48p"]
Nov 03 22:47:06.902395 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:06.902372    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2d6076-d576997dd-rh48p"]
Nov 03 22:47:06.904426 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:06.904400    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:47:08.091525 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:08.091490    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9749991e-8f89-488c-9f71-e7da0c122683" path="/var/lib/kubelet/pods/9749991e-8f89-488c-9f71-e7da0c122683/volumes"
Nov 03 22:47:08.389268 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:08.389156    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:47:08.758676 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:08.758644    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:47:08.906168 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:08.906140    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:47:09.963716 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:09.963342    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:47:11.400024 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:11.399977    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:47:12.873956 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:12.873913    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:47:14.442595 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:14.441762    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:47:15.888783 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:15.888730    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:47:17.426439 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:17.426405    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:47:18.800390 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:18.800357    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2d6076-5497f8bb66-mbzbx"]
Nov 03 22:47:18.800896 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:18.800585    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2d6076-5497f8bb66-mbzbx" podUID="23ec5be6-eefc-425a-a820-f639d9a0bcf3" containerName="maas-api" containerID="cri-o://8f3e8703bd1f2e18a8ada8eb055c5b0a9a3e723e9927b89b5406c2f66ae25cb0" gracePeriod=30
Nov 03 22:47:18.895943 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:18.895908    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:47:18.966265 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:18.966227    4431 generic.go:356] "Generic (PLEG): container finished" podID="23ec5be6-eefc-425a-a820-f639d9a0bcf3" containerID="8f3e8703bd1f2e18a8ada8eb055c5b0a9a3e723e9927b89b5406c2f66ae25cb0" exitCode=0
Nov 03 22:47:18.966265 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:18.966253    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2d6076-5497f8bb66-mbzbx" event={"ID":"23ec5be6-eefc-425a-a820-f639d9a0bcf3","Type":"ContainerDied","Data":"8f3e8703bd1f2e18a8ada8eb055c5b0a9a3e723e9927b89b5406c2f66ae25cb0"}
Nov 03 22:47:19.411325 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:19.411241    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:47:19.528408 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:19.528376    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2d6076-5497f8bb66-mbzbx"
Nov 03 22:47:19.557390 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:19.557349    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:47:19.712403 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:19.706756    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23ec5be6-eefc-425a-a820-f639d9a0bcf3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23ec5be6-eefc-425a-a820-f639d9a0bcf3-maas-api-tls\") pod \"23ec5be6-eefc-425a-a820-f639d9a0bcf3\" (UID: \"23ec5be6-eefc-425a-a820-f639d9a0bcf3\") "
Nov 03 22:47:19.712403 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:19.707852    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23ec5be6-eefc-425a-a820-f639d9a0bcf3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23ec5be6-eefc-425a-a820-f639d9a0bcf3-openshift-service-ca\") pod \"23ec5be6-eefc-425a-a820-f639d9a0bcf3\" (UID: \"23ec5be6-eefc-425a-a820-f639d9a0bcf3\") "
Nov 03 22:47:19.712403 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:19.707901    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23ec5be6-eefc-425a-a820-f639d9a0bcf3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/23ec5be6-eefc-425a-a820-f639d9a0bcf3-metrics-certs\") pod \"23ec5be6-eefc-425a-a820-f639d9a0bcf3\" (UID: \"23ec5be6-eefc-425a-a820-f639d9a0bcf3\") "
Nov 03 22:47:19.712403 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:19.708120    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23ec5be6-eefc-425a-a820-f639d9a0bcf3-kube-api-access-jmcfk\" (UniqueName: \"kubernetes.io/projected/23ec5be6-eefc-425a-a820-f639d9a0bcf3-kube-api-access-jmcfk\") pod \"23ec5be6-eefc-425a-a820-f639d9a0bcf3\" (UID: \"23ec5be6-eefc-425a-a820-f639d9a0bcf3\") "
Nov 03 22:47:19.712403 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:19.708924    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:47:19.712403 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:19.710226    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23ec5be6-eefc-425a-a820-f639d9a0bcf3-openshift-service-ca" pod "23ec5be6-eefc-425a-a820-f639d9a0bcf3" (UID: "23ec5be6-eefc-425a-a820-f639d9a0bcf3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:47:19.715719 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:19.713875    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23ec5be6-eefc-425a-a820-f639d9a0bcf3-kube-api-access-jmcfk" pod "23ec5be6-eefc-425a-a820-f639d9a0bcf3" (UID: "23ec5be6-eefc-425a-a820-f639d9a0bcf3"). InnerVolumeSpecName "kube-api-access-jmcfk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:47:19.716771 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:19.716557    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23ec5be6-eefc-425a-a820-f639d9a0bcf3-maas-api-tls" pod "23ec5be6-eefc-425a-a820-f639d9a0bcf3" (UID: "23ec5be6-eefc-425a-a820-f639d9a0bcf3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:47:19.716771 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:19.716627    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23ec5be6-eefc-425a-a820-f639d9a0bcf3-metrics-certs" pod "23ec5be6-eefc-425a-a820-f639d9a0bcf3" (UID: "23ec5be6-eefc-425a-a820-f639d9a0bcf3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:47:19.810713 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:19.810681    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmcfk\" (UniqueName: \"kubernetes.io/projected/23ec5be6-eefc-425a-a820-f639d9a0bcf3-kube-api-access-jmcfk\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:47:19.810713 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:19.810712    4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23ec5be6-eefc-425a-a820-f639d9a0bcf3-maas-api-tls\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:47:19.810713 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:19.810721    4431 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23ec5be6-eefc-425a-a820-f639d9a0bcf3-openshift-service-ca\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:47:19.811159 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:19.810730    4431 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23ec5be6-eefc-425a-a820-f639d9a0bcf3-metrics-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:47:19.976190 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:19.976107    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2d6076-5497f8bb66-mbzbx"
Nov 03 22:47:19.976190 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:19.976140    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2d6076-5497f8bb66-mbzbx" event={"ID":"23ec5be6-eefc-425a-a820-f639d9a0bcf3","Type":"ContainerDied","Data":"72bfc86c9972e9cc5b795169623e19be11e87346d5134920d7a1450a5fc45315"}
Nov 03 22:47:19.976412 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:19.976206    4431 scope.go:122] "RemoveContainer" containerID="8f3e8703bd1f2e18a8ada8eb055c5b0a9a3e723e9927b89b5406c2f66ae25cb0"
Nov 03 22:47:20.035635 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:20.035600    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2d6076-5497f8bb66-mbzbx"]
Nov 03 22:47:20.039178 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:20.039152    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2d6076-5497f8bb66-mbzbx"]
Nov 03 22:47:20.050903 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:20.050875    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:47:20.089429 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:20.089400    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23ec5be6-eefc-425a-a820-f639d9a0bcf3" path="/var/lib/kubelet/pods/23ec5be6-eefc-425a-a820-f639d9a0bcf3/volumes"
Nov 03 22:47:21.296449 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:21.296411    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:47:22.821224 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:22.821189    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:47:24.489557 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:24.489519    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:47:25.999292 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:25.999250    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:47:26.854402 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:26.854363    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:47:27.014750 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:27.014710    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:47:27.249261 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:27.243090    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:47:27.948913 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:27.948880    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:47:29.577311 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:29.577278    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:47:29.854747 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:29.854645    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:47:31.265051 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:47:31.265010    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:47:31.390164 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:31.390122    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:47:32.657318 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:32.657283    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:47:32.738723 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:32.738686    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp"]
Nov 03 22:47:32.738982 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:32.738943    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp" podUID="a1787f2c-7d0d-4bda-9a6d-f63a41b6b503" containerName="istio-proxy" containerID="cri-o://dc68de302c9f48dea33d7402dfac812486e5bb73bca24fa5852ec8b7bb84b123" gracePeriod=30
Nov 03 22:47:32.805128 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:32.805102    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:47:33.050895 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:33.050850    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:47:33.246112 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:33.246074    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t"]
Nov 03 22:47:33.246370 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:33.246340    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t" podUID="e3d85fcc-b84d-48a3-944a-810ce94062a2" containerName="istio-proxy" containerID="cri-o://d10817a4ba8e04585139df549ff3a090c6dcbce256e774d93b060b76fb534f92" gracePeriod=30
Nov 03 22:47:34.221182 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:34.221144    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-d6hr4"]
Nov 03 22:47:34.221660 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:34.221352    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-d6hr4" podUID="9286b874-939f-479a-b771-a4e877cbd50f" containerName="istio-proxy" containerID="cri-o://d2a10f47e05277e53af0a2e01ed45f476a16b28cfbe7b7cc8098a7a81e9362da" gracePeriod=30
Nov 03 22:47:34.241179 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:34.241144    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-crcsv"]
Nov 03 22:47:34.241713 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:34.241665    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-crcsv" podUID="3f5907c4-80f0-4055-b561-30650efcbb92" containerName="istio-proxy" containerID="cri-o://e37736308e2700ce41775f54017c9d0088d0960d98acc5d0bcf3c586deafed68" gracePeriod=30
Nov 03 22:47:34.825165 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:34.825126    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:47:34.909250 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:34.909197    4431 patch_prober.go:29] interesting pod/e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 22:47:34.909428 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:34.909292    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp" podUID="a1787f2c-7d0d-4bda-9a6d-f63a41b6b503" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 22:47:35.669667 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:35.669602    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:47:35.815285 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:35.815251    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:47:35.993606 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:35.993513    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-d4b6bb-75d4df64b8-nvlff"]
Nov 03 22:47:36.000014 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:35.998329    4431 cpu_manager.go:414] "containerMap: removing container" podUID="9749991e-8f89-488c-9f71-e7da0c122683" containerName="payload-processing"
Nov 03 22:47:36.000432 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.000411    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9749991e-8f89-488c-9f71-e7da0c122683" containerName="payload-processing"
Nov 03 22:47:36.000612 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.000600    4431 cpu_manager.go:414] "containerMap: removing container" podUID="892720cc-753b-43d6-ad3b-0494dff25623" containerName="istio-proxy"
Nov 03 22:47:36.000722 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.000711    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="892720cc-753b-43d6-ad3b-0494dff25623" containerName="istio-proxy"
Nov 03 22:47:36.000826 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.000816    4431 cpu_manager.go:414] "containerMap: removing container" podUID="57ada6e6-75cf-47a4-87d6-8ad44c0e442e" containerName="cleanup"
Nov 03 22:47:36.000915 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.000905    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57ada6e6-75cf-47a4-87d6-8ad44c0e442e" containerName="cleanup"
Nov 03 22:47:36.001015 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.001005    4431 cpu_manager.go:414] "containerMap: removing container" podUID="7ef57864-5184-4b21-8fae-201006734abf" containerName="cleanup"
Nov 03 22:47:36.001109 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.001098    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ef57864-5184-4b21-8fae-201006734abf" containerName="cleanup"
Nov 03 22:47:36.001201 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.001191    4431 cpu_manager.go:414] "containerMap: removing container" podUID="57ada6e6-75cf-47a4-87d6-8ad44c0e442e" containerName="cleanup"
Nov 03 22:47:36.001318 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.001305    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57ada6e6-75cf-47a4-87d6-8ad44c0e442e" containerName="cleanup"
Nov 03 22:47:36.001437 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.001427    4431 cpu_manager.go:414] "containerMap: removing container" podUID="0d201ce0-fde0-494b-988a-b19d31dbbcda" containerName="cleanup"
Nov 03 22:47:36.001533 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.001522    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d201ce0-fde0-494b-988a-b19d31dbbcda" containerName="cleanup"
Nov 03 22:47:36.001620 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.001610    4431 cpu_manager.go:414] "containerMap: removing container" podUID="8b5f2047-8624-42fb-9881-6d81c25867ee" containerName="cleanup"
Nov 03 22:47:36.001699 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.001689    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b5f2047-8624-42fb-9881-6d81c25867ee" containerName="cleanup"
Nov 03 22:47:36.002089 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.002074    4431 cpu_manager.go:414] "containerMap: removing container" podUID="cc558639-1e15-4789-aaae-d6535a4e62e8" containerName="cleanup"
Nov 03 22:47:36.002192 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.002180    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc558639-1e15-4789-aaae-d6535a4e62e8" containerName="cleanup"
Nov 03 22:47:36.002318 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.002307    4431 cpu_manager.go:414] "containerMap: removing container" podUID="7ef57864-5184-4b21-8fae-201006734abf" containerName="cleanup"
Nov 03 22:47:36.002397 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.002387    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ef57864-5184-4b21-8fae-201006734abf" containerName="cleanup"
Nov 03 22:47:36.002480 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.002470    4431 cpu_manager.go:414] "containerMap: removing container" podUID="4b6f92d6-3fbb-48fe-854f-6b5b899f56d1" containerName="cleanup"
Nov 03 22:47:36.002551 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.002540    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b6f92d6-3fbb-48fe-854f-6b5b899f56d1" containerName="cleanup"
Nov 03 22:47:36.002646 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.002636    4431 cpu_manager.go:414] "containerMap: removing container" podUID="8af62636-7952-421c-8302-0af2cfefe3b7" containerName="istio-proxy"
Nov 03 22:47:36.002747 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.002735    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8af62636-7952-421c-8302-0af2cfefe3b7" containerName="istio-proxy"
Nov 03 22:47:36.002835 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.002825    4431 cpu_manager.go:414] "containerMap: removing container" podUID="f511dde2-8f9b-4670-bd73-b7ee34d69706" containerName="cleanup"
Nov 03 22:47:36.002916 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.002905    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f511dde2-8f9b-4670-bd73-b7ee34d69706" containerName="cleanup"
Nov 03 22:47:36.002999 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.002990    4431 cpu_manager.go:414] "containerMap: removing container" podUID="f511dde2-8f9b-4670-bd73-b7ee34d69706" containerName="cleanup"
Nov 03 22:47:36.003074 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.003064    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f511dde2-8f9b-4670-bd73-b7ee34d69706" containerName="cleanup"
Nov 03 22:47:36.003154 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.003144    4431 cpu_manager.go:414] "containerMap: removing container" podUID="f9a74dcb-40e3-415a-a090-3619177efa8a" containerName="cleanup"
Nov 03 22:47:36.003247 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.003232    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9a74dcb-40e3-415a-a090-3619177efa8a" containerName="cleanup"
Nov 03 22:47:36.003340 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.003330    4431 cpu_manager.go:414] "containerMap: removing container" podUID="23ec5be6-eefc-425a-a820-f639d9a0bcf3" containerName="maas-api"
Nov 03 22:47:36.003435 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.003409    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23ec5be6-eefc-425a-a820-f639d9a0bcf3" containerName="maas-api"
Nov 03 22:47:36.003521 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.003511    4431 cpu_manager.go:414] "containerMap: removing container" podUID="f9a74dcb-40e3-415a-a090-3619177efa8a" containerName="cleanup"
Nov 03 22:47:36.003590 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.003580    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9a74dcb-40e3-415a-a090-3619177efa8a" containerName="cleanup"
Nov 03 22:47:36.003673 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.003663    4431 cpu_manager.go:414] "containerMap: removing container" podUID="4b6f92d6-3fbb-48fe-854f-6b5b899f56d1" containerName="cleanup"
Nov 03 22:47:36.003745 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.003735    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b6f92d6-3fbb-48fe-854f-6b5b899f56d1" containerName="cleanup"
Nov 03 22:47:36.003814 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.003804    4431 cpu_manager.go:414] "containerMap: removing container" podUID="0d201ce0-fde0-494b-988a-b19d31dbbcda" containerName="cleanup"
Nov 03 22:47:36.003894 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.003883    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d201ce0-fde0-494b-988a-b19d31dbbcda" containerName="cleanup"
Nov 03 22:47:36.003982 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.003972    4431 cpu_manager.go:414] "containerMap: removing container" podUID="4b6f92d6-3fbb-48fe-854f-6b5b899f56d1" containerName="cleanup"
Nov 03 22:47:36.004131 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.004117    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b6f92d6-3fbb-48fe-854f-6b5b899f56d1" containerName="cleanup"
Nov 03 22:47:36.004235 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.004209    4431 cpu_manager.go:414] "containerMap: removing container" podUID="10b331d4-81e4-4243-a645-69d95958db00" containerName="istio-proxy"
Nov 03 22:47:36.004488 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.004471    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10b331d4-81e4-4243-a645-69d95958db00" containerName="istio-proxy"
Nov 03 22:47:36.004607 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.004595    4431 cpu_manager.go:414] "containerMap: removing container" podUID="96676e8f-4e63-42e0-88b3-5588c9c2fed3" containerName="cleanup"
Nov 03 22:47:36.004701 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.004689    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96676e8f-4e63-42e0-88b3-5588c9c2fed3" containerName="cleanup"
Nov 03 22:47:36.004803 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.004791    4431 cpu_manager.go:414] "containerMap: removing container" podUID="0d201ce0-fde0-494b-988a-b19d31dbbcda" containerName="cleanup"
Nov 03 22:47:36.032928 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.004869    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d201ce0-fde0-494b-988a-b19d31dbbcda" containerName="cleanup"
Nov 03 22:47:36.032928 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.004890    4431 cpu_manager.go:414] "containerMap: removing container" podUID="8b5f2047-8624-42fb-9881-6d81c25867ee" containerName="cleanup"
Nov 03 22:47:36.032928 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.004897    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b5f2047-8624-42fb-9881-6d81c25867ee" containerName="cleanup"
Nov 03 22:47:36.032928 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.004910    4431 cpu_manager.go:414] "containerMap: removing container" podUID="cc558639-1e15-4789-aaae-d6535a4e62e8" containerName="cleanup"
Nov 03 22:47:36.032928 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.004917    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc558639-1e15-4789-aaae-d6535a4e62e8" containerName="cleanup"
Nov 03 22:47:36.032928 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.004934    4431 cpu_manager.go:414] "containerMap: removing container" podUID="96676e8f-4e63-42e0-88b3-5588c9c2fed3" containerName="cleanup"
Nov 03 22:47:36.032928 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.004941    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96676e8f-4e63-42e0-88b3-5588c9c2fed3" containerName="cleanup"
Nov 03 22:47:36.032928 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.004957    4431 cpu_manager.go:414] "containerMap: removing container" podUID="10b331d4-81e4-4243-a645-69d95958db00" containerName="istio-proxy"
Nov 03 22:47:36.032928 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.004963    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10b331d4-81e4-4243-a645-69d95958db00" containerName="istio-proxy"
Nov 03 22:47:36.032928 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.004980    4431 cpu_manager.go:414] "containerMap: removing container" podUID="8b5f2047-8624-42fb-9881-6d81c25867ee" containerName="cleanup"
Nov 03 22:47:36.032928 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.004987    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b5f2047-8624-42fb-9881-6d81c25867ee" containerName="cleanup"
Nov 03 22:47:36.032928 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.005001    4431 cpu_manager.go:414] "containerMap: removing container" podUID="96676e8f-4e63-42e0-88b3-5588c9c2fed3" containerName="cleanup"
Nov 03 22:47:36.032928 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.005009    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96676e8f-4e63-42e0-88b3-5588c9c2fed3" containerName="cleanup"
Nov 03 22:47:36.032928 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.005044    4431 cpu_manager.go:414] "containerMap: removing container" podUID="600936df-e54c-429b-ad28-e30a6a76af8f" containerName="cleanup"
Nov 03 22:47:36.032928 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.005052    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="600936df-e54c-429b-ad28-e30a6a76af8f" containerName="cleanup"
Nov 03 22:47:36.032928 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.005061    4431 cpu_manager.go:414] "containerMap: removing container" podUID="600936df-e54c-429b-ad28-e30a6a76af8f" containerName="cleanup"
Nov 03 22:47:36.032928 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.005068    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="600936df-e54c-429b-ad28-e30a6a76af8f" containerName="cleanup"
Nov 03 22:47:36.032928 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.005084    4431 cpu_manager.go:414] "containerMap: removing container" podUID="600936df-e54c-429b-ad28-e30a6a76af8f" containerName="cleanup"
Nov 03 22:47:36.032928 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.005092    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="600936df-e54c-429b-ad28-e30a6a76af8f" containerName="cleanup"
Nov 03 22:47:36.032928 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.005114    4431 cpu_manager.go:414] "containerMap: removing container" podUID="7dde620e-e87f-4f87-984e-6358a642a100" containerName="istio-proxy"
Nov 03 22:47:36.032928 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.005121    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dde620e-e87f-4f87-984e-6358a642a100" containerName="istio-proxy"
Nov 03 22:47:36.032928 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.005137    4431 cpu_manager.go:414] "containerMap: removing container" podUID="f9a74dcb-40e3-415a-a090-3619177efa8a" containerName="cleanup"
Nov 03 22:47:36.032928 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.005144    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9a74dcb-40e3-415a-a090-3619177efa8a" containerName="cleanup"
Nov 03 22:47:36.032928 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.005380    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="8af62636-7952-421c-8302-0af2cfefe3b7" containerName="istio-proxy"
Nov 03 22:47:36.032928 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.005396    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="f511dde2-8f9b-4670-bd73-b7ee34d69706" containerName="cleanup"
Nov 03 22:47:36.032928 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.005406    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="f511dde2-8f9b-4670-bd73-b7ee34d69706" containerName="cleanup"
Nov 03 22:47:36.032928 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.005417    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9a74dcb-40e3-415a-a090-3619177efa8a" containerName="cleanup"
Nov 03 22:47:36.032928 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.005425    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="23ec5be6-eefc-425a-a820-f639d9a0bcf3" containerName="maas-api"
Nov 03 22:47:36.032928 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.005435    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9a74dcb-40e3-415a-a090-3619177efa8a" containerName="cleanup"
Nov 03 22:47:36.032928 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.005445    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ef57864-5184-4b21-8fae-201006734abf" containerName="cleanup"
Nov 03 22:47:36.032928 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.005454    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b6f92d6-3fbb-48fe-854f-6b5b899f56d1" containerName="cleanup"
Nov 03 22:47:36.032928 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.005462    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d201ce0-fde0-494b-988a-b19d31dbbcda" containerName="cleanup"
Nov 03 22:47:36.032928 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.005470    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b6f92d6-3fbb-48fe-854f-6b5b899f56d1" containerName="cleanup"
Nov 03 22:47:36.032928 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.005479    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="10b331d4-81e4-4243-a645-69d95958db00" containerName="istio-proxy"
Nov 03 22:47:36.032928 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.005487    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="96676e8f-4e63-42e0-88b3-5588c9c2fed3" containerName="cleanup"
Nov 03 22:47:36.032928 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.005495    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d201ce0-fde0-494b-988a-b19d31dbbcda" containerName="cleanup"
Nov 03 22:47:36.032928 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.005505    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b5f2047-8624-42fb-9881-6d81c25867ee" containerName="cleanup"
Nov 03 22:47:36.032928 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.005513    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc558639-1e15-4789-aaae-d6535a4e62e8" containerName="cleanup"
Nov 03 22:47:36.032928 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.005522    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="96676e8f-4e63-42e0-88b3-5588c9c2fed3" containerName="cleanup"
Nov 03 22:47:36.032928 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.005530    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="57ada6e6-75cf-47a4-87d6-8ad44c0e442e" containerName="cleanup"
Nov 03 22:47:36.032928 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.005540    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b5f2047-8624-42fb-9881-6d81c25867ee" containerName="cleanup"
Nov 03 22:47:36.032928 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.005549    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="96676e8f-4e63-42e0-88b3-5588c9c2fed3" containerName="cleanup"
Nov 03 22:47:36.032928 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.005559    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="600936df-e54c-429b-ad28-e30a6a76af8f" containerName="cleanup"
Nov 03 22:47:36.032928 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.005568    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="600936df-e54c-429b-ad28-e30a6a76af8f" containerName="cleanup"
Nov 03 22:47:36.032928 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.005578    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="600936df-e54c-429b-ad28-e30a6a76af8f" containerName="cleanup"
Nov 03 22:47:36.032928 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.005589    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dde620e-e87f-4f87-984e-6358a642a100" containerName="istio-proxy"
Nov 03 22:47:36.032928 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.005599    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="9749991e-8f89-488c-9f71-e7da0c122683" containerName="payload-processing"
Nov 03 22:47:36.032928 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.005608    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="892720cc-753b-43d6-ad3b-0494dff25623" containerName="istio-proxy"
Nov 03 22:47:36.032928 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.005616    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="57ada6e6-75cf-47a4-87d6-8ad44c0e442e" containerName="cleanup"
Nov 03 22:47:36.032928 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.005626    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d201ce0-fde0-494b-988a-b19d31dbbcda" containerName="cleanup"
Nov 03 22:47:36.032928 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.005637    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc558639-1e15-4789-aaae-d6535a4e62e8" containerName="cleanup"
Nov 03 22:47:36.032928 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.005646    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ef57864-5184-4b21-8fae-201006734abf" containerName="cleanup"
Nov 03 22:47:36.066583 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.050956    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d4b6bb-75d4df64b8-nvlff"
Nov 03 22:47:36.092771 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.079805    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-d4b6bb\""
Nov 03 22:47:36.092771 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.079898    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-d4b6bb-dockercfg-fddmr\""
Nov 03 22:47:36.097618 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.097558    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60d398e2-9692-4010-83ed-d660486bc982-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-d4b6bb-75d4df64b8-nvlff\" (UID: \"60d398e2-9692-4010-83ed-d660486bc982\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d4b6bb-75d4df64b8-nvlff"
Nov 03 22:47:36.098457 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.098425    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9bd9\" (UniqueName: \"kubernetes.io/projected/60d398e2-9692-4010-83ed-d660486bc982-kube-api-access-b9bd9\") pod \"payload-processing-e2e-shared-a-w2-d4b6bb-75d4df64b8-nvlff\" (UID: \"60d398e2-9692-4010-83ed-d660486bc982\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d4b6bb-75d4df64b8-nvlff"
Nov 03 22:47:36.119881 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.119463    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-d4b6bb-75d4df64b8-nvlff"]
Nov 03 22:47:36.200693 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.200654    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60d398e2-9692-4010-83ed-d660486bc982-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-d4b6bb-75d4df64b8-nvlff\" (UID: \"60d398e2-9692-4010-83ed-d660486bc982\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d4b6bb-75d4df64b8-nvlff"
Nov 03 22:47:36.200872 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.200854    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9bd9\" (UniqueName: \"kubernetes.io/projected/60d398e2-9692-4010-83ed-d660486bc982-kube-api-access-b9bd9\") pod \"payload-processing-e2e-shared-a-w2-d4b6bb-75d4df64b8-nvlff\" (UID: \"60d398e2-9692-4010-83ed-d660486bc982\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d4b6bb-75d4df64b8-nvlff"
Nov 03 22:47:36.201390 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.201369    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60d398e2-9692-4010-83ed-d660486bc982-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-d4b6bb-75d4df64b8-nvlff\" (UID: \"60d398e2-9692-4010-83ed-d660486bc982\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d4b6bb-75d4df64b8-nvlff"
Nov 03 22:47:36.229747 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.229722    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9bd9\" (UniqueName: \"kubernetes.io/projected/60d398e2-9692-4010-83ed-d660486bc982-kube-api-access-b9bd9\") pod \"payload-processing-e2e-shared-a-w2-d4b6bb-75d4df64b8-nvlff\" (UID: \"60d398e2-9692-4010-83ed-d660486bc982\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d4b6bb-75d4df64b8-nvlff"
Nov 03 22:47:36.398806 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.398730    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d4b6bb-75d4df64b8-nvlff"
Nov 03 22:47:36.595261 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.595200    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:47:36.706512 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:36.706478    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-d4b6bb-75d4df64b8-nvlff"]
Nov 03 22:47:36.708393 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:47:36.708361    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60d398e2_9692_4010_83ed_d660486bc982.slice/crio-9e0ad0f542941a5c51461ef61e3674f44b7cccac80b4f46ff98d889cbee7bc1b WatchSource:0}: Error finding container 9e0ad0f542941a5c51461ef61e3674f44b7cccac80b4f46ff98d889cbee7bc1b: Status 404 returned error can't find the container with id 9e0ad0f542941a5c51461ef61e3674f44b7cccac80b4f46ff98d889cbee7bc1b
Nov 03 22:47:37.179589 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:37.179556    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d4b6bb-75d4df64b8-nvlff" event={"ID":"60d398e2-9692-4010-83ed-d660486bc982","Type":"ContainerStarted","Data":"5434b5113a9b565afc150107bdccb9debcebfdeb0e7f548cb2348a82dd4929c8"}
Nov 03 22:47:37.179778 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:37.179604    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d4b6bb-75d4df64b8-nvlff" event={"ID":"60d398e2-9692-4010-83ed-d660486bc982","Type":"ContainerStarted","Data":"9e0ad0f542941a5c51461ef61e3674f44b7cccac80b4f46ff98d889cbee7bc1b"}
Nov 03 22:47:37.179778 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:37.179652    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d4b6bb-75d4df64b8-nvlff"
Nov 03 22:47:37.225691 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:37.225634    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d4b6bb-75d4df64b8-nvlff" podStartSLOduration=2.225615008 podStartE2EDuration="2.225615008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:47:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:47:37.222546115 +0000 UTC m=+2081.248401978" watchObservedRunningTime="2026-11-03 22:47:37.225615008 +0000 UTC m=+2081.251470872"
Nov 03 22:47:38.015587 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:38.015525    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp"
Nov 03 22:47:38.161720 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:38.161685    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1787f2c-7d0d-4bda-9a6d-f63a41b6b503-kube-api-access-xwcck\" (UniqueName: \"kubernetes.io/projected/a1787f2c-7d0d-4bda-9a6d-f63a41b6b503-kube-api-access-xwcck\") pod \"a1787f2c-7d0d-4bda-9a6d-f63a41b6b503\" (UID: \"a1787f2c-7d0d-4bda-9a6d-f63a41b6b503\") "
Nov 03 22:47:38.161888 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:38.161813    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1787f2c-7d0d-4bda-9a6d-f63a41b6b503-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1787f2c-7d0d-4bda-9a6d-f63a41b6b503-workload-socket\") pod \"a1787f2c-7d0d-4bda-9a6d-f63a41b6b503\" (UID: \"a1787f2c-7d0d-4bda-9a6d-f63a41b6b503\") "
Nov 03 22:47:38.161888 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:38.161853    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1787f2c-7d0d-4bda-9a6d-f63a41b6b503-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1787f2c-7d0d-4bda-9a6d-f63a41b6b503-credential-socket\") pod \"a1787f2c-7d0d-4bda-9a6d-f63a41b6b503\" (UID: \"a1787f2c-7d0d-4bda-9a6d-f63a41b6b503\") "
Nov 03 22:47:38.161888 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:38.161886    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1787f2c-7d0d-4bda-9a6d-f63a41b6b503-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1787f2c-7d0d-4bda-9a6d-f63a41b6b503-istiod-ca-cert\") pod \"a1787f2c-7d0d-4bda-9a6d-f63a41b6b503\" (UID: \"a1787f2c-7d0d-4bda-9a6d-f63a41b6b503\") "
Nov 03 22:47:38.162025 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:38.161915    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1787f2c-7d0d-4bda-9a6d-f63a41b6b503-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1787f2c-7d0d-4bda-9a6d-f63a41b6b503-workload-certs\") pod \"a1787f2c-7d0d-4bda-9a6d-f63a41b6b503\" (UID: \"a1787f2c-7d0d-4bda-9a6d-f63a41b6b503\") "
Nov 03 22:47:38.162025 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:38.161966    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1787f2c-7d0d-4bda-9a6d-f63a41b6b503-istio-token\" (UniqueName: \"kubernetes.io/projected/a1787f2c-7d0d-4bda-9a6d-f63a41b6b503-istio-token\") pod \"a1787f2c-7d0d-4bda-9a6d-f63a41b6b503\" (UID: \"a1787f2c-7d0d-4bda-9a6d-f63a41b6b503\") "
Nov 03 22:47:38.162025 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:38.161992    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a1787f2c-7d0d-4bda-9a6d-f63a41b6b503-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1787f2c-7d0d-4bda-9a6d-f63a41b6b503-istio-podinfo\") pod \"a1787f2c-7d0d-4bda-9a6d-f63a41b6b503\" (UID: \"a1787f2c-7d0d-4bda-9a6d-f63a41b6b503\") "
Nov 03 22:47:38.162179 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:38.162055    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1787f2c-7d0d-4bda-9a6d-f63a41b6b503-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1787f2c-7d0d-4bda-9a6d-f63a41b6b503-istio-envoy\") pod \"a1787f2c-7d0d-4bda-9a6d-f63a41b6b503\" (UID: \"a1787f2c-7d0d-4bda-9a6d-f63a41b6b503\") "
Nov 03 22:47:38.162179 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:38.162131    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1787f2c-7d0d-4bda-9a6d-f63a41b6b503-workload-socket" pod "a1787f2c-7d0d-4bda-9a6d-f63a41b6b503" (UID: "a1787f2c-7d0d-4bda-9a6d-f63a41b6b503"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:47:38.162179 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:38.162154    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1787f2c-7d0d-4bda-9a6d-f63a41b6b503-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1787f2c-7d0d-4bda-9a6d-f63a41b6b503-istio-data\") pod \"a1787f2c-7d0d-4bda-9a6d-f63a41b6b503\" (UID: \"a1787f2c-7d0d-4bda-9a6d-f63a41b6b503\") "
Nov 03 22:47:38.162371 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:38.162269    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1787f2c-7d0d-4bda-9a6d-f63a41b6b503-workload-certs" pod "a1787f2c-7d0d-4bda-9a6d-f63a41b6b503" (UID: "a1787f2c-7d0d-4bda-9a6d-f63a41b6b503"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:47:38.162437 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:38.162382    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1787f2c-7d0d-4bda-9a6d-f63a41b6b503-istio-data" pod "a1787f2c-7d0d-4bda-9a6d-f63a41b6b503" (UID: "a1787f2c-7d0d-4bda-9a6d-f63a41b6b503"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:47:38.162437 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:38.162403    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1787f2c-7d0d-4bda-9a6d-f63a41b6b503-credential-socket" pod "a1787f2c-7d0d-4bda-9a6d-f63a41b6b503" (UID: "a1787f2c-7d0d-4bda-9a6d-f63a41b6b503"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:47:38.162437 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:38.162421    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1787f2c-7d0d-4bda-9a6d-f63a41b6b503-istiod-ca-cert" pod "a1787f2c-7d0d-4bda-9a6d-f63a41b6b503" (UID: "a1787f2c-7d0d-4bda-9a6d-f63a41b6b503"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:47:38.163037 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:38.163009    4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1787f2c-7d0d-4bda-9a6d-f63a41b6b503-workload-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:47:38.163157 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:38.163039    4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1787f2c-7d0d-4bda-9a6d-f63a41b6b503-credential-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:47:38.163157 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:38.163055    4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1787f2c-7d0d-4bda-9a6d-f63a41b6b503-istiod-ca-cert\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:47:38.163157 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:38.163069    4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1787f2c-7d0d-4bda-9a6d-f63a41b6b503-workload-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:47:38.163157 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:38.163081    4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1787f2c-7d0d-4bda-9a6d-f63a41b6b503-istio-data\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:47:38.164279 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:38.164194    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1787f2c-7d0d-4bda-9a6d-f63a41b6b503-istio-envoy" pod "a1787f2c-7d0d-4bda-9a6d-f63a41b6b503" (UID: "a1787f2c-7d0d-4bda-9a6d-f63a41b6b503"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:47:38.164279 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:38.164267    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1787f2c-7d0d-4bda-9a6d-f63a41b6b503-istio-token" pod "a1787f2c-7d0d-4bda-9a6d-f63a41b6b503" (UID: "a1787f2c-7d0d-4bda-9a6d-f63a41b6b503"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:47:38.164427 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:38.164289    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1787f2c-7d0d-4bda-9a6d-f63a41b6b503-kube-api-access-xwcck" pod "a1787f2c-7d0d-4bda-9a6d-f63a41b6b503" (UID: "a1787f2c-7d0d-4bda-9a6d-f63a41b6b503"). InnerVolumeSpecName "kube-api-access-xwcck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:47:38.164546 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:38.164530    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a1787f2c-7d0d-4bda-9a6d-f63a41b6b503-istio-podinfo" pod "a1787f2c-7d0d-4bda-9a6d-f63a41b6b503" (UID: "a1787f2c-7d0d-4bda-9a6d-f63a41b6b503"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 22:47:38.187478 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:38.187451    4431 generic.go:356] "Generic (PLEG): container finished" podID="a1787f2c-7d0d-4bda-9a6d-f63a41b6b503" containerID="dc68de302c9f48dea33d7402dfac812486e5bb73bca24fa5852ec8b7bb84b123" exitCode=0
Nov 03 22:47:38.187605 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:38.187505    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp"
Nov 03 22:47:38.187605 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:38.187537    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp" event={"ID":"a1787f2c-7d0d-4bda-9a6d-f63a41b6b503","Type":"ContainerDied","Data":"dc68de302c9f48dea33d7402dfac812486e5bb73bca24fa5852ec8b7bb84b123"}
Nov 03 22:47:38.187605 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:38.187589    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp" event={"ID":"a1787f2c-7d0d-4bda-9a6d-f63a41b6b503","Type":"ContainerDied","Data":"ec14cf58b0de06ddd7beff974f36f1acac21cd7b346fa5d5493155adebeae496"}
Nov 03 22:47:38.187775 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:38.187610    4431 scope.go:122] "RemoveContainer" containerID="dc68de302c9f48dea33d7402dfac812486e5bb73bca24fa5852ec8b7bb84b123"
Nov 03 22:47:38.199210 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:38.199189    4431 scope.go:122] "RemoveContainer" containerID="dc68de302c9f48dea33d7402dfac812486e5bb73bca24fa5852ec8b7bb84b123"
Nov 03 22:47:38.199551 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:47:38.199533    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc68de302c9f48dea33d7402dfac812486e5bb73bca24fa5852ec8b7bb84b123\": container with ID starting with dc68de302c9f48dea33d7402dfac812486e5bb73bca24fa5852ec8b7bb84b123 not found: ID does not exist" containerID="dc68de302c9f48dea33d7402dfac812486e5bb73bca24fa5852ec8b7bb84b123"
Nov 03 22:47:38.199608 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:38.199558    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc68de302c9f48dea33d7402dfac812486e5bb73bca24fa5852ec8b7bb84b123"} err="failed to get container status \"dc68de302c9f48dea33d7402dfac812486e5bb73bca24fa5852ec8b7bb84b123\": rpc error: code = NotFound desc = could not find container \"dc68de302c9f48dea33d7402dfac812486e5bb73bca24fa5852ec8b7bb84b123\": container with ID starting with dc68de302c9f48dea33d7402dfac812486e5bb73bca24fa5852ec8b7bb84b123 not found: ID does not exist"
Nov 03 22:47:38.230226 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:38.230186    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:47:38.258236 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:38.257896    4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-crcsv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 22:47:38.258236 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:38.257952    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-crcsv" podUID="3f5907c4-80f0-4055-b561-30650efcbb92" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 22:47:38.274382 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:38.274346    4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1787f2c-7d0d-4bda-9a6d-f63a41b6b503-istio-podinfo\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:47:38.274382 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:38.274387    4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1787f2c-7d0d-4bda-9a6d-f63a41b6b503-istio-envoy\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:47:38.274584 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:38.274408    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwcck\" (UniqueName: \"kubernetes.io/projected/a1787f2c-7d0d-4bda-9a6d-f63a41b6b503-kube-api-access-xwcck\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:47:38.274584 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:38.274421    4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1787f2c-7d0d-4bda-9a6d-f63a41b6b503-istio-token\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:47:38.276417 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:38.276378    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp"]
Nov 03 22:47:38.287999 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:38.287959    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-scbfp"]
Nov 03 22:47:38.839089 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:38.839048    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:47:39.026294 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.006091    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:47:39.198710 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.198674    4431 generic.go:356] "Generic (PLEG): container finished" podID="e3d85fcc-b84d-48a3-944a-810ce94062a2" containerID="d10817a4ba8e04585139df549ff3a090c6dcbce256e774d93b060b76fb534f92" exitCode=0
Nov 03 22:47:39.198859 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.198756    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t" event={"ID":"e3d85fcc-b84d-48a3-944a-810ce94062a2","Type":"ContainerDied","Data":"d10817a4ba8e04585139df549ff3a090c6dcbce256e774d93b060b76fb534f92"}
Nov 03 22:47:39.198859 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.198821    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t" event={"ID":"e3d85fcc-b84d-48a3-944a-810ce94062a2","Type":"ContainerDied","Data":"e39377b09b5364991d9dafeb33fdcfb58084089c35b348170ef5a4602670275c"}
Nov 03 22:47:39.198859 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.198847    4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e39377b09b5364991d9dafeb33fdcfb58084089c35b348170ef5a4602670275c"
Nov 03 22:47:39.217940 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.217913    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t"
Nov 03 22:47:39.318776 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.318737    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3d85fcc-b84d-48a3-944a-810ce94062a2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3d85fcc-b84d-48a3-944a-810ce94062a2-istio-data\") pod \"e3d85fcc-b84d-48a3-944a-810ce94062a2\" (UID: \"e3d85fcc-b84d-48a3-944a-810ce94062a2\") "
Nov 03 22:47:39.319032 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.318997    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3d85fcc-b84d-48a3-944a-810ce94062a2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3d85fcc-b84d-48a3-944a-810ce94062a2-credential-socket\") pod \"e3d85fcc-b84d-48a3-944a-810ce94062a2\" (UID: \"e3d85fcc-b84d-48a3-944a-810ce94062a2\") "
Nov 03 22:47:39.319129 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.319089    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3d85fcc-b84d-48a3-944a-810ce94062a2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3d85fcc-b84d-48a3-944a-810ce94062a2-workload-socket\") pod \"e3d85fcc-b84d-48a3-944a-810ce94062a2\" (UID: \"e3d85fcc-b84d-48a3-944a-810ce94062a2\") "
Nov 03 22:47:39.319192 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.319167    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3d85fcc-b84d-48a3-944a-810ce94062a2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3d85fcc-b84d-48a3-944a-810ce94062a2-workload-certs\") pod \"e3d85fcc-b84d-48a3-944a-810ce94062a2\" (UID: \"e3d85fcc-b84d-48a3-944a-810ce94062a2\") "
Nov 03 22:47:39.320067 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.320025    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3d85fcc-b84d-48a3-944a-810ce94062a2-istio-data" pod "e3d85fcc-b84d-48a3-944a-810ce94062a2" (UID: "e3d85fcc-b84d-48a3-944a-810ce94062a2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:47:39.320544 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.320522    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3d85fcc-b84d-48a3-944a-810ce94062a2-workload-socket" pod "e3d85fcc-b84d-48a3-944a-810ce94062a2" (UID: "e3d85fcc-b84d-48a3-944a-810ce94062a2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:47:39.320890 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.320868    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3d85fcc-b84d-48a3-944a-810ce94062a2-workload-certs" pod "e3d85fcc-b84d-48a3-944a-810ce94062a2" (UID: "e3d85fcc-b84d-48a3-944a-810ce94062a2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:47:39.321064 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.321043    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3d85fcc-b84d-48a3-944a-810ce94062a2-credential-socket" pod "e3d85fcc-b84d-48a3-944a-810ce94062a2" (UID: "e3d85fcc-b84d-48a3-944a-810ce94062a2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:47:39.321172 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.319205    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3d85fcc-b84d-48a3-944a-810ce94062a2-kube-api-access-k28v6\" (UniqueName: \"kubernetes.io/projected/e3d85fcc-b84d-48a3-944a-810ce94062a2-kube-api-access-k28v6\") pod \"e3d85fcc-b84d-48a3-944a-810ce94062a2\" (UID: \"e3d85fcc-b84d-48a3-944a-810ce94062a2\") "
Nov 03 22:47:39.321374 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.321355    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e3d85fcc-b84d-48a3-944a-810ce94062a2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3d85fcc-b84d-48a3-944a-810ce94062a2-istio-podinfo\") pod \"e3d85fcc-b84d-48a3-944a-810ce94062a2\" (UID: \"e3d85fcc-b84d-48a3-944a-810ce94062a2\") "
Nov 03 22:47:39.321500 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.321485    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3d85fcc-b84d-48a3-944a-810ce94062a2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3d85fcc-b84d-48a3-944a-810ce94062a2-istiod-ca-cert\") pod \"e3d85fcc-b84d-48a3-944a-810ce94062a2\" (UID: \"e3d85fcc-b84d-48a3-944a-810ce94062a2\") "
Nov 03 22:47:39.321630 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.321615    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3d85fcc-b84d-48a3-944a-810ce94062a2-istio-token\" (UniqueName: \"kubernetes.io/projected/e3d85fcc-b84d-48a3-944a-810ce94062a2-istio-token\") pod \"e3d85fcc-b84d-48a3-944a-810ce94062a2\" (UID: \"e3d85fcc-b84d-48a3-944a-810ce94062a2\") "
Nov 03 22:47:39.321853 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.321835    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3d85fcc-b84d-48a3-944a-810ce94062a2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3d85fcc-b84d-48a3-944a-810ce94062a2-istio-envoy\") pod \"e3d85fcc-b84d-48a3-944a-810ce94062a2\" (UID: \"e3d85fcc-b84d-48a3-944a-810ce94062a2\") "
Nov 03 22:47:39.323023 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.322995    4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3d85fcc-b84d-48a3-944a-810ce94062a2-istio-data\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:47:39.323135 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.323027    4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3d85fcc-b84d-48a3-944a-810ce94062a2-credential-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:47:39.323135 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.323041    4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3d85fcc-b84d-48a3-944a-810ce94062a2-workload-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:47:39.323135 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.323056    4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3d85fcc-b84d-48a3-944a-810ce94062a2-workload-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:47:39.330009 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.329034    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3d85fcc-b84d-48a3-944a-810ce94062a2-istiod-ca-cert" pod "e3d85fcc-b84d-48a3-944a-810ce94062a2" (UID: "e3d85fcc-b84d-48a3-944a-810ce94062a2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:47:39.337590 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.337564    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3d85fcc-b84d-48a3-944a-810ce94062a2-istio-envoy" pod "e3d85fcc-b84d-48a3-944a-810ce94062a2" (UID: "e3d85fcc-b84d-48a3-944a-810ce94062a2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:47:39.339195 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.339169    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3d85fcc-b84d-48a3-944a-810ce94062a2-istio-token" pod "e3d85fcc-b84d-48a3-944a-810ce94062a2" (UID: "e3d85fcc-b84d-48a3-944a-810ce94062a2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:47:39.340759 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.340734    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3d85fcc-b84d-48a3-944a-810ce94062a2-kube-api-access-k28v6" pod "e3d85fcc-b84d-48a3-944a-810ce94062a2" (UID: "e3d85fcc-b84d-48a3-944a-810ce94062a2"). InnerVolumeSpecName "kube-api-access-k28v6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:47:39.341016 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.340997    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e3d85fcc-b84d-48a3-944a-810ce94062a2-istio-podinfo" pod "e3d85fcc-b84d-48a3-944a-810ce94062a2" (UID: "e3d85fcc-b84d-48a3-944a-810ce94062a2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 22:47:39.427259 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.425032    4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3d85fcc-b84d-48a3-944a-810ce94062a2-istio-podinfo\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:47:39.427259 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.425075    4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3d85fcc-b84d-48a3-944a-810ce94062a2-istiod-ca-cert\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:47:39.427259 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.425089    4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3d85fcc-b84d-48a3-944a-810ce94062a2-istio-token\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:47:39.427259 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.425102    4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3d85fcc-b84d-48a3-944a-810ce94062a2-istio-envoy\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:47:39.427259 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.425117    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k28v6\" (UniqueName: \"kubernetes.io/projected/e3d85fcc-b84d-48a3-944a-810ce94062a2-kube-api-access-k28v6\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:47:39.531808 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.523095    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-d6hr4"
Nov 03 22:47:39.531969 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.531944    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9286b874-939f-479a-b771-a4e877cbd50f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9286b874-939f-479a-b771-a4e877cbd50f-workload-socket\") pod \"9286b874-939f-479a-b771-a4e877cbd50f\" (UID: \"9286b874-939f-479a-b771-a4e877cbd50f\") "
Nov 03 22:47:39.532894 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.532855    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9286b874-939f-479a-b771-a4e877cbd50f-workload-socket" pod "9286b874-939f-479a-b771-a4e877cbd50f" (UID: "9286b874-939f-479a-b771-a4e877cbd50f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:47:39.586030 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.584919    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-crcsv"
Nov 03 22:47:39.633636 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.633599    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9286b874-939f-479a-b771-a4e877cbd50f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9286b874-939f-479a-b771-a4e877cbd50f-istio-envoy\") pod \"9286b874-939f-479a-b771-a4e877cbd50f\" (UID: \"9286b874-939f-479a-b771-a4e877cbd50f\") "
Nov 03 22:47:39.633822 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.633649    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9286b874-939f-479a-b771-a4e877cbd50f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9286b874-939f-479a-b771-a4e877cbd50f-credential-socket\") pod \"9286b874-939f-479a-b771-a4e877cbd50f\" (UID: \"9286b874-939f-479a-b771-a4e877cbd50f\") "
Nov 03 22:47:39.633822 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.633792    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f5907c4-80f0-4055-b561-30650efcbb92-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f5907c4-80f0-4055-b561-30650efcbb92-istiod-ca-cert\") pod \"3f5907c4-80f0-4055-b561-30650efcbb92\" (UID: \"3f5907c4-80f0-4055-b561-30650efcbb92\") "
Nov 03 22:47:39.633957 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.633895    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f5907c4-80f0-4055-b561-30650efcbb92-kube-api-access-pktfd\" (UniqueName: \"kubernetes.io/projected/3f5907c4-80f0-4055-b561-30650efcbb92-kube-api-access-pktfd\") pod \"3f5907c4-80f0-4055-b561-30650efcbb92\" (UID: \"3f5907c4-80f0-4055-b561-30650efcbb92\") "
Nov 03 22:47:39.633957 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.633938    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f5907c4-80f0-4055-b561-30650efcbb92-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f5907c4-80f0-4055-b561-30650efcbb92-workload-socket\") pod \"3f5907c4-80f0-4055-b561-30650efcbb92\" (UID: \"3f5907c4-80f0-4055-b561-30650efcbb92\") "
Nov 03 22:47:39.634069 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.633999    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f5907c4-80f0-4055-b561-30650efcbb92-istio-token\" (UniqueName: \"kubernetes.io/projected/3f5907c4-80f0-4055-b561-30650efcbb92-istio-token\") pod \"3f5907c4-80f0-4055-b561-30650efcbb92\" (UID: \"3f5907c4-80f0-4055-b561-30650efcbb92\") "
Nov 03 22:47:39.634069 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.634046    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f5907c4-80f0-4055-b561-30650efcbb92-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f5907c4-80f0-4055-b561-30650efcbb92-credential-socket\") pod \"3f5907c4-80f0-4055-b561-30650efcbb92\" (UID: \"3f5907c4-80f0-4055-b561-30650efcbb92\") "
Nov 03 22:47:39.634269 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.634056    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9286b874-939f-479a-b771-a4e877cbd50f-credential-socket" pod "9286b874-939f-479a-b771-a4e877cbd50f" (UID: "9286b874-939f-479a-b771-a4e877cbd50f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:47:39.634269 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.634109    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9286b874-939f-479a-b771-a4e877cbd50f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9286b874-939f-479a-b771-a4e877cbd50f-workload-certs\") pod \"9286b874-939f-479a-b771-a4e877cbd50f\" (UID: \"9286b874-939f-479a-b771-a4e877cbd50f\") "
Nov 03 22:47:39.634269 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.634158    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f5907c4-80f0-4055-b561-30650efcbb92-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f5907c4-80f0-4055-b561-30650efcbb92-istio-envoy\") pod \"3f5907c4-80f0-4055-b561-30650efcbb92\" (UID: \"3f5907c4-80f0-4055-b561-30650efcbb92\") "
Nov 03 22:47:39.634454 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.634194    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f5907c4-80f0-4055-b561-30650efcbb92-istiod-ca-cert" pod "3f5907c4-80f0-4055-b561-30650efcbb92" (UID: "3f5907c4-80f0-4055-b561-30650efcbb92"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:47:39.634454 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.634255    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3f5907c4-80f0-4055-b561-30650efcbb92-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f5907c4-80f0-4055-b561-30650efcbb92-istio-podinfo\") pod \"3f5907c4-80f0-4055-b561-30650efcbb92\" (UID: \"3f5907c4-80f0-4055-b561-30650efcbb92\") "
Nov 03 22:47:39.634454 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.634310    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f5907c4-80f0-4055-b561-30650efcbb92-workload-socket" pod "3f5907c4-80f0-4055-b561-30650efcbb92" (UID: "3f5907c4-80f0-4055-b561-30650efcbb92"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:47:39.634454 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.634354    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f5907c4-80f0-4055-b561-30650efcbb92-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f5907c4-80f0-4055-b561-30650efcbb92-istio-data\") pod \"3f5907c4-80f0-4055-b561-30650efcbb92\" (UID: \"3f5907c4-80f0-4055-b561-30650efcbb92\") "
Nov 03 22:47:39.634454 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.634390    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9286b874-939f-479a-b771-a4e877cbd50f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9286b874-939f-479a-b771-a4e877cbd50f-istiod-ca-cert\") pod \"9286b874-939f-479a-b771-a4e877cbd50f\" (UID: \"9286b874-939f-479a-b771-a4e877cbd50f\") "
Nov 03 22:47:39.634728 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.634475    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f5907c4-80f0-4055-b561-30650efcbb92-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f5907c4-80f0-4055-b561-30650efcbb92-workload-certs\") pod \"3f5907c4-80f0-4055-b561-30650efcbb92\" (UID: \"3f5907c4-80f0-4055-b561-30650efcbb92\") "
Nov 03 22:47:39.634728 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.634515    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9286b874-939f-479a-b771-a4e877cbd50f-workload-certs" pod "9286b874-939f-479a-b771-a4e877cbd50f" (UID: "9286b874-939f-479a-b771-a4e877cbd50f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:47:39.634728 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.634550    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9286b874-939f-479a-b771-a4e877cbd50f-istio-token\" (UniqueName: \"kubernetes.io/projected/9286b874-939f-479a-b771-a4e877cbd50f-istio-token\") pod \"9286b874-939f-479a-b771-a4e877cbd50f\" (UID: \"9286b874-939f-479a-b771-a4e877cbd50f\") "
Nov 03 22:47:39.634728 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.634584    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9286b874-939f-479a-b771-a4e877cbd50f-kube-api-access-hf9ft\" (UniqueName: \"kubernetes.io/projected/9286b874-939f-479a-b771-a4e877cbd50f-kube-api-access-hf9ft\") pod \"9286b874-939f-479a-b771-a4e877cbd50f\" (UID: \"9286b874-939f-479a-b771-a4e877cbd50f\") "
Nov 03 22:47:39.634728 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.634620    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9286b874-939f-479a-b771-a4e877cbd50f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9286b874-939f-479a-b771-a4e877cbd50f-istio-podinfo\") pod \"9286b874-939f-479a-b771-a4e877cbd50f\" (UID: \"9286b874-939f-479a-b771-a4e877cbd50f\") "
Nov 03 22:47:39.634728 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.634684    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9286b874-939f-479a-b771-a4e877cbd50f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9286b874-939f-479a-b771-a4e877cbd50f-istio-data\") pod \"9286b874-939f-479a-b771-a4e877cbd50f\" (UID: \"9286b874-939f-479a-b771-a4e877cbd50f\") "
Nov 03 22:47:39.635054 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.634873    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f5907c4-80f0-4055-b561-30650efcbb92-credential-socket" pod "3f5907c4-80f0-4055-b561-30650efcbb92" (UID: "3f5907c4-80f0-4055-b561-30650efcbb92"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:47:39.635160 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.635126    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f5907c4-80f0-4055-b561-30650efcbb92-workload-certs" pod "3f5907c4-80f0-4055-b561-30650efcbb92" (UID: "3f5907c4-80f0-4055-b561-30650efcbb92"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:47:39.635411 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.635384    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9286b874-939f-479a-b771-a4e877cbd50f-istio-data" pod "9286b874-939f-479a-b771-a4e877cbd50f" (UID: "9286b874-939f-479a-b771-a4e877cbd50f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:47:39.635732 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.635711    4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f5907c4-80f0-4055-b561-30650efcbb92-workload-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:47:39.635732 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.635726    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f5907c4-80f0-4055-b561-30650efcbb92-istio-data" pod "3f5907c4-80f0-4055-b561-30650efcbb92" (UID: "3f5907c4-80f0-4055-b561-30650efcbb92"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:47:39.635903 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.635739    4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9286b874-939f-479a-b771-a4e877cbd50f-istio-data\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:47:39.635903 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.635752    4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9286b874-939f-479a-b771-a4e877cbd50f-credential-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:47:39.635903 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.635766    4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f5907c4-80f0-4055-b561-30650efcbb92-istiod-ca-cert\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:47:39.635903 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.635780    4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9286b874-939f-479a-b771-a4e877cbd50f-workload-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:47:39.635903 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.635795    4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f5907c4-80f0-4055-b561-30650efcbb92-workload-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:47:39.635903 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.635810    4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f5907c4-80f0-4055-b561-30650efcbb92-credential-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:47:39.635903 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.635822    4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9286b874-939f-479a-b771-a4e877cbd50f-workload-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:47:39.636636 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.636158    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9286b874-939f-479a-b771-a4e877cbd50f-istiod-ca-cert" pod "9286b874-939f-479a-b771-a4e877cbd50f" (UID: "9286b874-939f-479a-b771-a4e877cbd50f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:47:39.636636 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.636298    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9286b874-939f-479a-b771-a4e877cbd50f-istio-envoy" pod "9286b874-939f-479a-b771-a4e877cbd50f" (UID: "9286b874-939f-479a-b771-a4e877cbd50f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:47:39.643492 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.637730    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f5907c4-80f0-4055-b561-30650efcbb92-istio-envoy" pod "3f5907c4-80f0-4055-b561-30650efcbb92" (UID: "3f5907c4-80f0-4055-b561-30650efcbb92"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:47:39.643492 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.639406    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9286b874-939f-479a-b771-a4e877cbd50f-istio-token" pod "9286b874-939f-479a-b771-a4e877cbd50f" (UID: "9286b874-939f-479a-b771-a4e877cbd50f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:47:39.643492 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.639630    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f5907c4-80f0-4055-b561-30650efcbb92-kube-api-access-pktfd" pod "3f5907c4-80f0-4055-b561-30650efcbb92" (UID: "3f5907c4-80f0-4055-b561-30650efcbb92"). InnerVolumeSpecName "kube-api-access-pktfd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:47:39.643492 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.640743    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f5907c4-80f0-4055-b561-30650efcbb92-istio-token" pod "3f5907c4-80f0-4055-b561-30650efcbb92" (UID: "3f5907c4-80f0-4055-b561-30650efcbb92"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:47:39.643492 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.641158    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9286b874-939f-479a-b771-a4e877cbd50f-kube-api-access-hf9ft" pod "9286b874-939f-479a-b771-a4e877cbd50f" (UID: "9286b874-939f-479a-b771-a4e877cbd50f"). InnerVolumeSpecName "kube-api-access-hf9ft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:47:39.643492 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.642138    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9286b874-939f-479a-b771-a4e877cbd50f-istio-podinfo" pod "9286b874-939f-479a-b771-a4e877cbd50f" (UID: "9286b874-939f-479a-b771-a4e877cbd50f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 22:47:39.644771 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.644729    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f5907c4-80f0-4055-b561-30650efcbb92-istio-podinfo" pod "3f5907c4-80f0-4055-b561-30650efcbb92" (UID: "3f5907c4-80f0-4055-b561-30650efcbb92"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 22:47:39.737091 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.737043    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hf9ft\" (UniqueName: \"kubernetes.io/projected/9286b874-939f-479a-b771-a4e877cbd50f-kube-api-access-hf9ft\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:47:39.737091 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.737085    4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9286b874-939f-479a-b771-a4e877cbd50f-istio-podinfo\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:47:39.737091 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.737096    4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9286b874-939f-479a-b771-a4e877cbd50f-istio-envoy\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:47:39.737371 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.737105    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pktfd\" (UniqueName: \"kubernetes.io/projected/3f5907c4-80f0-4055-b561-30650efcbb92-kube-api-access-pktfd\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:47:39.737371 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.737113    4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f5907c4-80f0-4055-b561-30650efcbb92-istio-token\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:47:39.737371 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.737121    4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f5907c4-80f0-4055-b561-30650efcbb92-istio-envoy\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:47:39.737371 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.737129    4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f5907c4-80f0-4055-b561-30650efcbb92-istio-podinfo\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:47:39.737371 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.737137    4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f5907c4-80f0-4055-b561-30650efcbb92-istio-data\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:47:39.737371 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.737144    4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9286b874-939f-479a-b771-a4e877cbd50f-istiod-ca-cert\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:47:39.737371 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:39.737151    4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9286b874-939f-479a-b771-a4e877cbd50f-istio-token\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:47:40.000786 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:40.000745    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:47:40.090605 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:40.090563    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1787f2c-7d0d-4bda-9a6d-f63a41b6b503" path="/var/lib/kubelet/pods/a1787f2c-7d0d-4bda-9a6d-f63a41b6b503/volumes"
Nov 03 22:47:40.207662 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:40.207628    4431 generic.go:356] "Generic (PLEG): container finished" podID="9286b874-939f-479a-b771-a4e877cbd50f" containerID="d2a10f47e05277e53af0a2e01ed45f476a16b28cfbe7b7cc8098a7a81e9362da" exitCode=0
Nov 03 22:47:40.207805 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:40.207681    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-d6hr4"
Nov 03 22:47:40.207805 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:40.207706    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-d6hr4" event={"ID":"9286b874-939f-479a-b771-a4e877cbd50f","Type":"ContainerDied","Data":"d2a10f47e05277e53af0a2e01ed45f476a16b28cfbe7b7cc8098a7a81e9362da"}
Nov 03 22:47:40.207805 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:40.207764    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-d6hr4" event={"ID":"9286b874-939f-479a-b771-a4e877cbd50f","Type":"ContainerDied","Data":"bb802e136c290b589cbfa360117486ef138c0c8275e7bd692bf303562b0702ab"}
Nov 03 22:47:40.207805 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:40.207787    4431 scope.go:122] "RemoveContainer" containerID="d2a10f47e05277e53af0a2e01ed45f476a16b28cfbe7b7cc8098a7a81e9362da"
Nov 03 22:47:40.209632 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:40.209605    4431 generic.go:356] "Generic (PLEG): container finished" podID="3f5907c4-80f0-4055-b561-30650efcbb92" containerID="e37736308e2700ce41775f54017c9d0088d0960d98acc5d0bcf3c586deafed68" exitCode=0
Nov 03 22:47:40.209768 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:40.209649    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-crcsv"
Nov 03 22:47:40.209768 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:40.209711    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-crcsv" event={"ID":"3f5907c4-80f0-4055-b561-30650efcbb92","Type":"ContainerDied","Data":"e37736308e2700ce41775f54017c9d0088d0960d98acc5d0bcf3c586deafed68"}
Nov 03 22:47:40.209768 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:40.209760    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-crcsv" event={"ID":"3f5907c4-80f0-4055-b561-30650efcbb92","Type":"ContainerDied","Data":"c2a70ba29b46f05fc27459b96a3d8d740f47fea5c5d2d94f28d643b43668cd3a"}
Nov 03 22:47:40.209931 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:40.209875    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t"
Nov 03 22:47:40.220904 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:40.220825    4431 scope.go:122] "RemoveContainer" containerID="d2a10f47e05277e53af0a2e01ed45f476a16b28cfbe7b7cc8098a7a81e9362da"
Nov 03 22:47:40.221455 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:47:40.221328    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2a10f47e05277e53af0a2e01ed45f476a16b28cfbe7b7cc8098a7a81e9362da\": container with ID starting with d2a10f47e05277e53af0a2e01ed45f476a16b28cfbe7b7cc8098a7a81e9362da not found: ID does not exist" containerID="d2a10f47e05277e53af0a2e01ed45f476a16b28cfbe7b7cc8098a7a81e9362da"
Nov 03 22:47:40.221455 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:40.221361    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2a10f47e05277e53af0a2e01ed45f476a16b28cfbe7b7cc8098a7a81e9362da"} err="failed to get container status \"d2a10f47e05277e53af0a2e01ed45f476a16b28cfbe7b7cc8098a7a81e9362da\": rpc error: code = NotFound desc = could not find container \"d2a10f47e05277e53af0a2e01ed45f476a16b28cfbe7b7cc8098a7a81e9362da\": container with ID starting with d2a10f47e05277e53af0a2e01ed45f476a16b28cfbe7b7cc8098a7a81e9362da not found: ID does not exist"
Nov 03 22:47:40.221455 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:40.221386    4431 scope.go:122] "RemoveContainer" containerID="e37736308e2700ce41775f54017c9d0088d0960d98acc5d0bcf3c586deafed68"
Nov 03 22:47:40.233447 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:40.233424    4431 scope.go:122] "RemoveContainer" containerID="e37736308e2700ce41775f54017c9d0088d0960d98acc5d0bcf3c586deafed68"
Nov 03 22:47:40.233814 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:47:40.233792    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e37736308e2700ce41775f54017c9d0088d0960d98acc5d0bcf3c586deafed68\": container with ID starting with e37736308e2700ce41775f54017c9d0088d0960d98acc5d0bcf3c586deafed68 not found: ID does not exist" containerID="e37736308e2700ce41775f54017c9d0088d0960d98acc5d0bcf3c586deafed68"
Nov 03 22:47:40.233895 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:40.233822    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e37736308e2700ce41775f54017c9d0088d0960d98acc5d0bcf3c586deafed68"} err="failed to get container status \"e37736308e2700ce41775f54017c9d0088d0960d98acc5d0bcf3c586deafed68\": rpc error: code = NotFound desc = could not find container \"e37736308e2700ce41775f54017c9d0088d0960d98acc5d0bcf3c586deafed68\": container with ID starting with e37736308e2700ce41775f54017c9d0088d0960d98acc5d0bcf3c586deafed68 not found: ID does not exist"
Nov 03 22:47:40.263283 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:40.263190    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-crcsv"]
Nov 03 22:47:40.266922 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:40.266895    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-crcsv"]
Nov 03 22:47:40.307184 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:40.307151    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-d6hr4"]
Nov 03 22:47:40.311352 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:40.311323    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-d6hr4"]
Nov 03 22:47:40.343246 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:40.343207    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t"]
Nov 03 22:47:40.351560 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:40.351530    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-m2b7t"]
Nov 03 22:47:41.783244 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:41.779341    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:47:42.089321 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:42.089242    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f5907c4-80f0-4055-b561-30650efcbb92" path="/var/lib/kubelet/pods/3f5907c4-80f0-4055-b561-30650efcbb92/volumes"
Nov 03 22:47:42.089770 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:42.089756    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9286b874-939f-479a-b771-a4e877cbd50f" path="/var/lib/kubelet/pods/9286b874-939f-479a-b771-a4e877cbd50f/volumes"
Nov 03 22:47:42.090198 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:42.090185    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3d85fcc-b84d-48a3-944a-810ce94062a2" path="/var/lib/kubelet/pods/e3d85fcc-b84d-48a3-944a-810ce94062a2/volumes"
Nov 03 22:47:43.495619 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:43.495583    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:47:44.198160 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:44.198121    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:47:44.342429 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:44.342394    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:47:44.958828 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:44.958788    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:47:46.519990 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:46.519956    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:47:48.116577 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:48.116540    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:47:48.191359 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:48.191326    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d4b6bb-75d4df64b8-nvlff"
Nov 03 22:47:48.375147 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:48.375072    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:47:49.598947 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:49.598906    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:47:49.742742 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:49.742703    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:47:51.149509 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:51.146662    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:47:52.747107 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:52.747045    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:47:53.467730 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:53.467692    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:47:54.218275 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:54.218207    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:47:55.706983 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:55.706930    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:47:56.762562 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:56.762530    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mx5q8_ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea/ovn-acl-logging/0.log"
Nov 03 22:47:56.773149 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:56.773124    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mx5q8_ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea/ovn-acl-logging/0.log"
Nov 03 22:47:57.321033 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:57.320992    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:47:58.572100 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:58.572067    4431 scope.go:122] "RemoveContainer" containerID="d10817a4ba8e04585139df549ff3a090c6dcbce256e774d93b060b76fb534f92"
Nov 03 22:47:59.447988 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:47:59.447942    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:48:00.912959 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:00.912898    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:48:01.260483 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:48:01.260383    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:48:02.393612 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:02.393574    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:48:03.614970 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:03.614935    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b1ec3a-6b45cb6cb5-d9lqz"]
Nov 03 22:48:03.615506 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:03.615192    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b1ec3a-6b45cb6cb5-d9lqz" podUID="a598ca79-668e-464c-ad39-5dd482218699" containerName="maas-api" containerID="cri-o://13f61122fea2fac0aa1e42efdae1d183277370ee963abb86473f3cc61af8b292" gracePeriod=30
Nov 03 22:48:03.731842 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:03.731809    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:48:04.226456 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:04.226428    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:48:04.245278 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:04.245252    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b1ec3a-6b45cb6cb5-d9lqz"
Nov 03 22:48:04.308882 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:04.308846    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a598ca79-668e-464c-ad39-5dd482218699-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a598ca79-668e-464c-ad39-5dd482218699-openshift-service-ca\") pod \"a598ca79-668e-464c-ad39-5dd482218699\" (UID: \"a598ca79-668e-464c-ad39-5dd482218699\") "
Nov 03 22:48:04.309055 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:04.308961    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a598ca79-668e-464c-ad39-5dd482218699-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a598ca79-668e-464c-ad39-5dd482218699-metrics-certs\") pod \"a598ca79-668e-464c-ad39-5dd482218699\" (UID: \"a598ca79-668e-464c-ad39-5dd482218699\") "
Nov 03 22:48:04.309198 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:04.309176    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a598ca79-668e-464c-ad39-5dd482218699-kube-api-access-nmcm7\" (UniqueName: \"kubernetes.io/projected/a598ca79-668e-464c-ad39-5dd482218699-kube-api-access-nmcm7\") pod \"a598ca79-668e-464c-ad39-5dd482218699\" (UID: \"a598ca79-668e-464c-ad39-5dd482218699\") "
Nov 03 22:48:04.309404 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:04.309380    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a598ca79-668e-464c-ad39-5dd482218699-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a598ca79-668e-464c-ad39-5dd482218699-maas-api-tls\") pod \"a598ca79-668e-464c-ad39-5dd482218699\" (UID: \"a598ca79-668e-464c-ad39-5dd482218699\") "
Nov 03 22:48:04.311335 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:04.311306    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a598ca79-668e-464c-ad39-5dd482218699-openshift-service-ca" pod "a598ca79-668e-464c-ad39-5dd482218699" (UID: "a598ca79-668e-464c-ad39-5dd482218699"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:48:04.315454 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:04.315428    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a598ca79-668e-464c-ad39-5dd482218699-kube-api-access-nmcm7" pod "a598ca79-668e-464c-ad39-5dd482218699" (UID: "a598ca79-668e-464c-ad39-5dd482218699"). InnerVolumeSpecName "kube-api-access-nmcm7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:48:04.315712 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:04.315683    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a598ca79-668e-464c-ad39-5dd482218699-maas-api-tls" pod "a598ca79-668e-464c-ad39-5dd482218699" (UID: "a598ca79-668e-464c-ad39-5dd482218699"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:48:04.320392 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:04.320367    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a598ca79-668e-464c-ad39-5dd482218699-metrics-certs" pod "a598ca79-668e-464c-ad39-5dd482218699" (UID: "a598ca79-668e-464c-ad39-5dd482218699"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:48:04.371618 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:04.371583    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:48:04.423409 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:04.423286    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmcm7\" (UniqueName: \"kubernetes.io/projected/a598ca79-668e-464c-ad39-5dd482218699-kube-api-access-nmcm7\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:48:04.423409 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:04.423323    4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a598ca79-668e-464c-ad39-5dd482218699-maas-api-tls\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:48:04.423409 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:04.423340    4431 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a598ca79-668e-464c-ad39-5dd482218699-openshift-service-ca\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:48:04.423409 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:04.423352    4431 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a598ca79-668e-464c-ad39-5dd482218699-metrics-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:48:04.433324 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:04.433292    4431 generic.go:356] "Generic (PLEG): container finished" podID="a598ca79-668e-464c-ad39-5dd482218699" containerID="13f61122fea2fac0aa1e42efdae1d183277370ee963abb86473f3cc61af8b292" exitCode=0
Nov 03 22:48:04.433475 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:04.433372    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b1ec3a-6b45cb6cb5-d9lqz"
Nov 03 22:48:04.433475 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:04.433371    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b1ec3a-6b45cb6cb5-d9lqz" event={"ID":"a598ca79-668e-464c-ad39-5dd482218699","Type":"ContainerDied","Data":"13f61122fea2fac0aa1e42efdae1d183277370ee963abb86473f3cc61af8b292"}
Nov 03 22:48:04.433475 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:04.433432    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b1ec3a-6b45cb6cb5-d9lqz" event={"ID":"a598ca79-668e-464c-ad39-5dd482218699","Type":"ContainerDied","Data":"bc2062aeb965932323f8ad35fb2ba79d118c390102475c2f7a9737331eae6d11"}
Nov 03 22:48:04.433475 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:04.433463    4431 scope.go:122] "RemoveContainer" containerID="13f61122fea2fac0aa1e42efdae1d183277370ee963abb86473f3cc61af8b292"
Nov 03 22:48:04.444868 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:04.444846    4431 scope.go:122] "RemoveContainer" containerID="13f61122fea2fac0aa1e42efdae1d183277370ee963abb86473f3cc61af8b292"
Nov 03 22:48:04.445137 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:48:04.445118    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13f61122fea2fac0aa1e42efdae1d183277370ee963abb86473f3cc61af8b292\": container with ID starting with 13f61122fea2fac0aa1e42efdae1d183277370ee963abb86473f3cc61af8b292 not found: ID does not exist" containerID="13f61122fea2fac0aa1e42efdae1d183277370ee963abb86473f3cc61af8b292"
Nov 03 22:48:04.445183 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:04.445146    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13f61122fea2fac0aa1e42efdae1d183277370ee963abb86473f3cc61af8b292"} err="failed to get container status \"13f61122fea2fac0aa1e42efdae1d183277370ee963abb86473f3cc61af8b292\": rpc error: code = NotFound desc = could not find container \"13f61122fea2fac0aa1e42efdae1d183277370ee963abb86473f3cc61af8b292\": container with ID starting with 13f61122fea2fac0aa1e42efdae1d183277370ee963abb86473f3cc61af8b292 not found: ID does not exist"
Nov 03 22:48:04.485752 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:04.485650    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b1ec3a-6b45cb6cb5-d9lqz"]
Nov 03 22:48:04.489591 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:04.489563    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b1ec3a-6b45cb6cb5-d9lqz"]
Nov 03 22:48:04.542823 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:04.542386    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:48:04.910194 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:04.910099    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:48:06.092192 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:06.092149    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a598ca79-668e-464c-ad39-5dd482218699" path="/var/lib/kubelet/pods/a598ca79-668e-464c-ad39-5dd482218699/volumes"
Nov 03 22:48:06.154096 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:06.135439    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:48:07.809085 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:07.809057    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:48:09.290498 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:09.290465    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:48:10.817199 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:10.817120    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:48:11.103365 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:11.103286    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:48:11.262748 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:11.262714    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:48:11.465894 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:11.465858    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:48:12.108888 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:12.108811    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:48:12.537145 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:12.537111    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:48:14.229331 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:14.229291    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:48:15.759566 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:15.759515    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d4b6bb-878757bb6-fxn94"]
Nov 03 22:48:15.760542 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:15.760519    4431 cpu_manager.go:414] "containerMap: removing container" podUID="cc558639-1e15-4789-aaae-d6535a4e62e8" containerName="cleanup"
Nov 03 22:48:15.760671 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:15.760544    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc558639-1e15-4789-aaae-d6535a4e62e8" containerName="cleanup"
Nov 03 22:48:15.760671 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:15.760566    4431 cpu_manager.go:414] "containerMap: removing container" podUID="9286b874-939f-479a-b771-a4e877cbd50f" containerName="istio-proxy"
Nov 03 22:48:15.760671 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:15.760574    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9286b874-939f-479a-b771-a4e877cbd50f" containerName="istio-proxy"
Nov 03 22:48:15.760671 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:15.760587    4431 cpu_manager.go:414] "containerMap: removing container" podUID="e3d85fcc-b84d-48a3-944a-810ce94062a2" containerName="istio-proxy"
Nov 03 22:48:15.760671 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:15.760594    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3d85fcc-b84d-48a3-944a-810ce94062a2" containerName="istio-proxy"
Nov 03 22:48:15.760671 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:15.760644    4431 cpu_manager.go:414] "containerMap: removing container" podUID="a598ca79-668e-464c-ad39-5dd482218699" containerName="maas-api"
Nov 03 22:48:15.760671 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:15.760651    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a598ca79-668e-464c-ad39-5dd482218699" containerName="maas-api"
Nov 03 22:48:15.760671 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:15.760658    4431 cpu_manager.go:414] "containerMap: removing container" podUID="57ada6e6-75cf-47a4-87d6-8ad44c0e442e" containerName="cleanup"
Nov 03 22:48:15.760671 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:15.760663    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57ada6e6-75cf-47a4-87d6-8ad44c0e442e" containerName="cleanup"
Nov 03 22:48:15.760671 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:15.760671    4431 cpu_manager.go:414] "containerMap: removing container" podUID="a1787f2c-7d0d-4bda-9a6d-f63a41b6b503" containerName="istio-proxy"
Nov 03 22:48:15.760671 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:15.760676    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1787f2c-7d0d-4bda-9a6d-f63a41b6b503" containerName="istio-proxy"
Nov 03 22:48:15.761120 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:15.760690    4431 cpu_manager.go:414] "containerMap: removing container" podUID="7ef57864-5184-4b21-8fae-201006734abf" containerName="cleanup"
Nov 03 22:48:15.761120 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:15.760695    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ef57864-5184-4b21-8fae-201006734abf" containerName="cleanup"
Nov 03 22:48:15.761120 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:15.760709    4431 cpu_manager.go:414] "containerMap: removing container" podUID="3f5907c4-80f0-4055-b561-30650efcbb92" containerName="istio-proxy"
Nov 03 22:48:15.761120 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:15.760714    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f5907c4-80f0-4055-b561-30650efcbb92" containerName="istio-proxy"
Nov 03 22:48:15.761120 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:15.760724    4431 cpu_manager.go:414] "containerMap: removing container" podUID="f511dde2-8f9b-4670-bd73-b7ee34d69706" containerName="cleanup"
Nov 03 22:48:15.761120 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:15.760729    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f511dde2-8f9b-4670-bd73-b7ee34d69706" containerName="cleanup"
Nov 03 22:48:15.761120 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:15.760862    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1787f2c-7d0d-4bda-9a6d-f63a41b6b503" containerName="istio-proxy"
Nov 03 22:48:15.761120 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:15.760872    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f5907c4-80f0-4055-b561-30650efcbb92" containerName="istio-proxy"
Nov 03 22:48:15.761120 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:15.760879    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="f511dde2-8f9b-4670-bd73-b7ee34d69706" containerName="cleanup"
Nov 03 22:48:15.761120 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:15.760885    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9a74dcb-40e3-415a-a090-3619177efa8a" containerName="cleanup"
Nov 03 22:48:15.761120 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:15.760890    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ef57864-5184-4b21-8fae-201006734abf" containerName="cleanup"
Nov 03 22:48:15.761120 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:15.760898    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b6f92d6-3fbb-48fe-854f-6b5b899f56d1" containerName="cleanup"
Nov 03 22:48:15.761120 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:15.760904    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="10b331d4-81e4-4243-a645-69d95958db00" containerName="istio-proxy"
Nov 03 22:48:15.761120 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:15.760910    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc558639-1e15-4789-aaae-d6535a4e62e8" containerName="cleanup"
Nov 03 22:48:15.761120 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:15.760915    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="9286b874-939f-479a-b771-a4e877cbd50f" containerName="istio-proxy"
Nov 03 22:48:15.761120 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:15.760921    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3d85fcc-b84d-48a3-944a-810ce94062a2" containerName="istio-proxy"
Nov 03 22:48:15.761120 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:15.760928    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b5f2047-8624-42fb-9881-6d81c25867ee" containerName="cleanup"
Nov 03 22:48:15.761120 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:15.760936    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="a598ca79-668e-464c-ad39-5dd482218699" containerName="maas-api"
Nov 03 22:48:15.761120 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:15.760940    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="57ada6e6-75cf-47a4-87d6-8ad44c0e442e" containerName="cleanup"
Nov 03 22:48:15.765834 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:15.765813    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d4b6bb-878757bb6-fxn94"
Nov 03 22:48:15.803751 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:15.803721    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-d4b6bb-serving-cert\""
Nov 03 22:48:15.807933 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:15.807906    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d4b6bb-878757bb6-fxn94"]
Nov 03 22:48:15.827378 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:15.827347    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-654tf\" (UniqueName: \"kubernetes.io/projected/bb8f978d-2eb9-4cf4-a7ea-ca3caf0c11ab-kube-api-access-654tf\") pod \"maas-api-e2e-shared-a-w2-d4b6bb-878757bb6-fxn94\" (UID: \"bb8f978d-2eb9-4cf4-a7ea-ca3caf0c11ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d4b6bb-878757bb6-fxn94"
Nov 03 22:48:15.827528 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:15.827409    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb8f978d-2eb9-4cf4-a7ea-ca3caf0c11ab-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-d4b6bb-878757bb6-fxn94\" (UID: \"bb8f978d-2eb9-4cf4-a7ea-ca3caf0c11ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d4b6bb-878757bb6-fxn94"
Nov 03 22:48:15.827619 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:15.827598    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb8f978d-2eb9-4cf4-a7ea-ca3caf0c11ab-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-d4b6bb-878757bb6-fxn94\" (UID: \"bb8f978d-2eb9-4cf4-a7ea-ca3caf0c11ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d4b6bb-878757bb6-fxn94"
Nov 03 22:48:15.827661 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:15.827635    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb8f978d-2eb9-4cf4-a7ea-ca3caf0c11ab-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-d4b6bb-878757bb6-fxn94\" (UID: \"bb8f978d-2eb9-4cf4-a7ea-ca3caf0c11ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d4b6bb-878757bb6-fxn94"
Nov 03 22:48:15.898759 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:15.898727    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:48:15.929336 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:15.929301    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb8f978d-2eb9-4cf4-a7ea-ca3caf0c11ab-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-d4b6bb-878757bb6-fxn94\" (UID: \"bb8f978d-2eb9-4cf4-a7ea-ca3caf0c11ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d4b6bb-878757bb6-fxn94"
Nov 03 22:48:15.929524 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:15.929347    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb8f978d-2eb9-4cf4-a7ea-ca3caf0c11ab-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-d4b6bb-878757bb6-fxn94\" (UID: \"bb8f978d-2eb9-4cf4-a7ea-ca3caf0c11ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d4b6bb-878757bb6-fxn94"
Nov 03 22:48:15.929524 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:15.929484    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-654tf\" (UniqueName: \"kubernetes.io/projected/bb8f978d-2eb9-4cf4-a7ea-ca3caf0c11ab-kube-api-access-654tf\") pod \"maas-api-e2e-shared-a-w2-d4b6bb-878757bb6-fxn94\" (UID: \"bb8f978d-2eb9-4cf4-a7ea-ca3caf0c11ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d4b6bb-878757bb6-fxn94"
Nov 03 22:48:15.929630 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:15.929529    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb8f978d-2eb9-4cf4-a7ea-ca3caf0c11ab-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-d4b6bb-878757bb6-fxn94\" (UID: \"bb8f978d-2eb9-4cf4-a7ea-ca3caf0c11ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d4b6bb-878757bb6-fxn94"
Nov 03 22:48:15.930175 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:15.930139    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb8f978d-2eb9-4cf4-a7ea-ca3caf0c11ab-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-d4b6bb-878757bb6-fxn94\" (UID: \"bb8f978d-2eb9-4cf4-a7ea-ca3caf0c11ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d4b6bb-878757bb6-fxn94"
Nov 03 22:48:15.931979 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:15.931958    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb8f978d-2eb9-4cf4-a7ea-ca3caf0c11ab-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-d4b6bb-878757bb6-fxn94\" (UID: \"bb8f978d-2eb9-4cf4-a7ea-ca3caf0c11ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d4b6bb-878757bb6-fxn94"
Nov 03 22:48:15.932128 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:15.932104    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb8f978d-2eb9-4cf4-a7ea-ca3caf0c11ab-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-d4b6bb-878757bb6-fxn94\" (UID: \"bb8f978d-2eb9-4cf4-a7ea-ca3caf0c11ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d4b6bb-878757bb6-fxn94"
Nov 03 22:48:15.952341 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:15.952304    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-654tf\" (UniqueName: \"kubernetes.io/projected/bb8f978d-2eb9-4cf4-a7ea-ca3caf0c11ab-kube-api-access-654tf\") pod \"maas-api-e2e-shared-a-w2-d4b6bb-878757bb6-fxn94\" (UID: \"bb8f978d-2eb9-4cf4-a7ea-ca3caf0c11ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d4b6bb-878757bb6-fxn94"
Nov 03 22:48:16.049681 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:16.049498    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:48:16.082753 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:16.082719    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d4b6bb-878757bb6-fxn94"
Nov 03 22:48:16.391490 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:16.391461    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d4b6bb-878757bb6-fxn94"]
Nov 03 22:48:16.392522 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:48:16.392496    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb8f978d_2eb9_4cf4_a7ea_ca3caf0c11ab.slice/crio-7ffc1e6d64e946a024886cb7dad6d726722e9fc813ac5d9e7d4cd0af5d634330 WatchSource:0}: Error finding container 7ffc1e6d64e946a024886cb7dad6d726722e9fc813ac5d9e7d4cd0af5d634330: Status 404 returned error can't find the container with id 7ffc1e6d64e946a024886cb7dad6d726722e9fc813ac5d9e7d4cd0af5d634330
Nov 03 22:48:16.394023 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:16.394002    4431 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 22:48:16.519283 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:16.519249    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d4b6bb-878757bb6-fxn94" event={"ID":"bb8f978d-2eb9-4cf4-a7ea-ca3caf0c11ab","Type":"ContainerStarted","Data":"7ffc1e6d64e946a024886cb7dad6d726722e9fc813ac5d9e7d4cd0af5d634330"}
Nov 03 22:48:17.529031 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:17.528980    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d4b6bb-878757bb6-fxn94" event={"ID":"bb8f978d-2eb9-4cf4-a7ea-ca3caf0c11ab","Type":"ContainerStarted","Data":"797d8772836441cbbdc4f350a6e3f5d26e2a4e7f1277ad18460d955ff9d9248b"}
Nov 03 22:48:17.529586 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:17.529054    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d4b6bb-878757bb6-fxn94"
Nov 03 22:48:17.570791 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:17.570735    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d4b6bb-878757bb6-fxn94" podStartSLOduration=2.212732693 podStartE2EDuration="2.570720807s" totalImagesPullingTime="357.988114ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:48:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:48:17.570259306 +0000 UTC m=+2121.596115169" watchObservedRunningTime="2026-11-03 22:48:17.570720807 +0000 UTC m=+2121.596576668"
Nov 03 22:48:17.575277 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:17.575250    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:48:19.232303 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:19.232268    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:48:20.805329 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:20.805290    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:48:22.359801 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:22.359741    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:48:22.607077 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:22.607043    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:48:22.743356 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:22.743302    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:48:23.542507 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:23.542477    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d4b6bb-878757bb6-fxn94"
Nov 03 22:48:23.973849 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:23.973804    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:48:24.916428 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:24.916393    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-71a7df/test-model-71a7df-kserve-6967965559-nd2vz"]
Nov 03 22:48:24.987408 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:24.987353    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:48:25.618526 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:25.618496    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:48:27.403246 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:27.403177    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:48:27.843095 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:27.843052    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2xmvs"]
Nov 03 22:48:27.843400 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:27.843354    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2xmvs" podUID="152e2bae-faf4-4d6f-9791-01fbbe4581c3" containerName="discovery" containerID="cri-o://e8f9f6e3d9852db86fb9c3d769e8f52e19eaf9377c52d1dc20b614588106c18c" gracePeriod=30
Nov 03 22:48:28.151093 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:28.151062    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2xmvs"
Nov 03 22:48:28.226006 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:28.225972    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/152e2bae-faf4-4d6f-9791-01fbbe4581c3-kube-api-access-rv7gg\" (UniqueName: \"kubernetes.io/projected/152e2bae-faf4-4d6f-9791-01fbbe4581c3-kube-api-access-rv7gg\") pod \"152e2bae-faf4-4d6f-9791-01fbbe4581c3\" (UID: \"152e2bae-faf4-4d6f-9791-01fbbe4581c3\") "
Nov 03 22:48:28.226179 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:28.226022    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/152e2bae-faf4-4d6f-9791-01fbbe4581c3-istio-token\" (UniqueName: \"kubernetes.io/projected/152e2bae-faf4-4d6f-9791-01fbbe4581c3-istio-token\") pod \"152e2bae-faf4-4d6f-9791-01fbbe4581c3\" (UID: \"152e2bae-faf4-4d6f-9791-01fbbe4581c3\") "
Nov 03 22:48:28.226179 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:28.226072    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/152e2bae-faf4-4d6f-9791-01fbbe4581c3-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/152e2bae-faf4-4d6f-9791-01fbbe4581c3-local-certs\") pod \"152e2bae-faf4-4d6f-9791-01fbbe4581c3\" (UID: \"152e2bae-faf4-4d6f-9791-01fbbe4581c3\") "
Nov 03 22:48:28.226179 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:28.226128    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/152e2bae-faf4-4d6f-9791-01fbbe4581c3-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/152e2bae-faf4-4d6f-9791-01fbbe4581c3-istio-kubeconfig\") pod \"152e2bae-faf4-4d6f-9791-01fbbe4581c3\" (UID: \"152e2bae-faf4-4d6f-9791-01fbbe4581c3\") "
Nov 03 22:48:28.226372 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:28.226197    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/152e2bae-faf4-4d6f-9791-01fbbe4581c3-cacerts\" (UniqueName: \"kubernetes.io/secret/152e2bae-faf4-4d6f-9791-01fbbe4581c3-cacerts\") pod \"152e2bae-faf4-4d6f-9791-01fbbe4581c3\" (UID: \"152e2bae-faf4-4d6f-9791-01fbbe4581c3\") "
Nov 03 22:48:28.226434 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:28.226370    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/152e2bae-faf4-4d6f-9791-01fbbe4581c3-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/152e2bae-faf4-4d6f-9791-01fbbe4581c3-istio-csr-dns-cert\") pod \"152e2bae-faf4-4d6f-9791-01fbbe4581c3\" (UID: \"152e2bae-faf4-4d6f-9791-01fbbe4581c3\") "
Nov 03 22:48:28.226792 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:28.226766    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/152e2bae-faf4-4d6f-9791-01fbbe4581c3-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/152e2bae-faf4-4d6f-9791-01fbbe4581c3-istio-csr-ca-configmap\") pod \"152e2bae-faf4-4d6f-9791-01fbbe4581c3\" (UID: \"152e2bae-faf4-4d6f-9791-01fbbe4581c3\") "
Nov 03 22:48:28.227455 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:28.227428    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/152e2bae-faf4-4d6f-9791-01fbbe4581c3-istio-csr-ca-configmap" pod "152e2bae-faf4-4d6f-9791-01fbbe4581c3" (UID: "152e2bae-faf4-4d6f-9791-01fbbe4581c3"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:48:28.228920 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:28.228894    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/152e2bae-faf4-4d6f-9791-01fbbe4581c3-cacerts" pod "152e2bae-faf4-4d6f-9791-01fbbe4581c3" (UID: "152e2bae-faf4-4d6f-9791-01fbbe4581c3"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:48:28.229035 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:28.228913    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/152e2bae-faf4-4d6f-9791-01fbbe4581c3-istio-kubeconfig" pod "152e2bae-faf4-4d6f-9791-01fbbe4581c3" (UID: "152e2bae-faf4-4d6f-9791-01fbbe4581c3"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:48:28.229035 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:28.228895    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/152e2bae-faf4-4d6f-9791-01fbbe4581c3-istio-token" pod "152e2bae-faf4-4d6f-9791-01fbbe4581c3" (UID: "152e2bae-faf4-4d6f-9791-01fbbe4581c3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:48:28.229173 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:28.229154    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/152e2bae-faf4-4d6f-9791-01fbbe4581c3-kube-api-access-rv7gg" pod "152e2bae-faf4-4d6f-9791-01fbbe4581c3" (UID: "152e2bae-faf4-4d6f-9791-01fbbe4581c3"). InnerVolumeSpecName "kube-api-access-rv7gg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:48:28.229282 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:28.229253    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/152e2bae-faf4-4d6f-9791-01fbbe4581c3-local-certs" pod "152e2bae-faf4-4d6f-9791-01fbbe4581c3" (UID: "152e2bae-faf4-4d6f-9791-01fbbe4581c3"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:48:28.229713 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:28.229693    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/152e2bae-faf4-4d6f-9791-01fbbe4581c3-istio-csr-dns-cert" pod "152e2bae-faf4-4d6f-9791-01fbbe4581c3" (UID: "152e2bae-faf4-4d6f-9791-01fbbe4581c3"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:48:28.329004 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:28.328969    4431 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/152e2bae-faf4-4d6f-9791-01fbbe4581c3-istio-kubeconfig\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:48:28.329004 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:28.329002    4431 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/152e2bae-faf4-4d6f-9791-01fbbe4581c3-cacerts\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:48:28.329004 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:28.329011    4431 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/152e2bae-faf4-4d6f-9791-01fbbe4581c3-istio-csr-dns-cert\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:48:28.329269 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:28.329020    4431 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/152e2bae-faf4-4d6f-9791-01fbbe4581c3-istio-csr-ca-configmap\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:48:28.329269 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:28.329030    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rv7gg\" (UniqueName: \"kubernetes.io/projected/152e2bae-faf4-4d6f-9791-01fbbe4581c3-kube-api-access-rv7gg\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:48:28.329269 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:28.329039    4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/152e2bae-faf4-4d6f-9791-01fbbe4581c3-istio-token\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:48:28.329269 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:28.329047    4431 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/152e2bae-faf4-4d6f-9791-01fbbe4581c3-local-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:48:28.629136 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:28.629097    4431 generic.go:356] "Generic (PLEG): container finished" podID="152e2bae-faf4-4d6f-9791-01fbbe4581c3" containerID="e8f9f6e3d9852db86fb9c3d769e8f52e19eaf9377c52d1dc20b614588106c18c" exitCode=0
Nov 03 22:48:28.629136 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:28.629148    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2xmvs"
Nov 03 22:48:28.629680 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:28.629187    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2xmvs" event={"ID":"152e2bae-faf4-4d6f-9791-01fbbe4581c3","Type":"ContainerDied","Data":"e8f9f6e3d9852db86fb9c3d769e8f52e19eaf9377c52d1dc20b614588106c18c"}
Nov 03 22:48:28.629680 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:28.629286    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2xmvs" event={"ID":"152e2bae-faf4-4d6f-9791-01fbbe4581c3","Type":"ContainerDied","Data":"8b085f3a1d9a577bc4a99b811dd41071da0ef3de39cbdb03f8e591bc472f7b32"}
Nov 03 22:48:28.629680 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:28.629320    4431 scope.go:122] "RemoveContainer" containerID="e8f9f6e3d9852db86fb9c3d769e8f52e19eaf9377c52d1dc20b614588106c18c"
Nov 03 22:48:28.641806 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:28.641789    4431 scope.go:122] "RemoveContainer" containerID="e8f9f6e3d9852db86fb9c3d769e8f52e19eaf9377c52d1dc20b614588106c18c"
Nov 03 22:48:28.642089 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:48:28.642067    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8f9f6e3d9852db86fb9c3d769e8f52e19eaf9377c52d1dc20b614588106c18c\": container with ID starting with e8f9f6e3d9852db86fb9c3d769e8f52e19eaf9377c52d1dc20b614588106c18c not found: ID does not exist" containerID="e8f9f6e3d9852db86fb9c3d769e8f52e19eaf9377c52d1dc20b614588106c18c"
Nov 03 22:48:28.642157 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:28.642102    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8f9f6e3d9852db86fb9c3d769e8f52e19eaf9377c52d1dc20b614588106c18c"} err="failed to get container status \"e8f9f6e3d9852db86fb9c3d769e8f52e19eaf9377c52d1dc20b614588106c18c\": rpc error: code = NotFound desc = could not find container \"e8f9f6e3d9852db86fb9c3d769e8f52e19eaf9377c52d1dc20b614588106c18c\": container with ID starting with e8f9f6e3d9852db86fb9c3d769e8f52e19eaf9377c52d1dc20b614588106c18c not found: ID does not exist"
Nov 03 22:48:28.683025 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:28.682994    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2xmvs"]
Nov 03 22:48:28.686585 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:28.686554    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2xmvs"]
Nov 03 22:48:29.123529 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:29.123499    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:48:30.090159 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:30.090115    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="152e2bae-faf4-4d6f-9791-01fbbe4581c3" path="/var/lib/kubelet/pods/152e2bae-faf4-4d6f-9791-01fbbe4581c3/volumes"
Nov 03 22:48:30.855101 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:30.855064    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:48:31.260261 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:48:31.260205    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:48:32.427666 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:32.427628    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:48:34.222937 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:34.222904    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:48:35.634286 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:35.634252    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9e540d-79df886dc8-q6kwt"]
Nov 03 22:48:35.814571 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:35.814536    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:48:35.965249 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:35.965186    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:48:36.124998 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:36.124966    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:48:37.631129 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:37.631099    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:48:39.135450 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:39.135416    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:48:39.953546 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:39.953509    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-71a7df/test-model-71a7df-kserve-6967965559-nd2vz" podUID="1018108a-ceed-4fd7-8ad8-23f7478d7cae" containerName="main" containerID="cri-o://f984a6dd41f37095f6790345dcbd010e24b408e84dac8664569844455323b1e0" gracePeriod=45
Nov 03 22:48:40.018803 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:40.018757    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-71a7df/test-model-71a7df-kserve-6967965559-nd2vz" podUID="1018108a-ceed-4fd7-8ad8-23f7478d7cae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": dial tcp 10.133.0.118:8000: connect: connection refused"
Nov 03 22:48:40.233170 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:40.233137    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-71a7df/test-model-71a7df-kserve-6967965559-nd2vz"
Nov 03 22:48:40.309400 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:40.309366    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1018108a-ceed-4fd7-8ad8-23f7478d7cae-tls-certs\" (UniqueName: \"kubernetes.io/secret/1018108a-ceed-4fd7-8ad8-23f7478d7cae-tls-certs\") pod \"1018108a-ceed-4fd7-8ad8-23f7478d7cae\" (UID: \"1018108a-ceed-4fd7-8ad8-23f7478d7cae\") "
Nov 03 22:48:40.309594 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:40.309474    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1018108a-ceed-4fd7-8ad8-23f7478d7cae-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1018108a-ceed-4fd7-8ad8-23f7478d7cae-model-cache\") pod \"1018108a-ceed-4fd7-8ad8-23f7478d7cae\" (UID: \"1018108a-ceed-4fd7-8ad8-23f7478d7cae\") "
Nov 03 22:48:40.309594 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:40.309508    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1018108a-ceed-4fd7-8ad8-23f7478d7cae-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1018108a-ceed-4fd7-8ad8-23f7478d7cae-dshm\") pod \"1018108a-ceed-4fd7-8ad8-23f7478d7cae\" (UID: \"1018108a-ceed-4fd7-8ad8-23f7478d7cae\") "
Nov 03 22:48:40.309594 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:40.309571    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1018108a-ceed-4fd7-8ad8-23f7478d7cae-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1018108a-ceed-4fd7-8ad8-23f7478d7cae-tmp-dir\") pod \"1018108a-ceed-4fd7-8ad8-23f7478d7cae\" (UID: \"1018108a-ceed-4fd7-8ad8-23f7478d7cae\") "
Nov 03 22:48:40.309809 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:40.309769    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1018108a-ceed-4fd7-8ad8-23f7478d7cae-model-cache" pod "1018108a-ceed-4fd7-8ad8-23f7478d7cae" (UID: "1018108a-ceed-4fd7-8ad8-23f7478d7cae"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:48:40.309880 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:40.309804    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1018108a-ceed-4fd7-8ad8-23f7478d7cae-home\" (UniqueName: \"kubernetes.io/empty-dir/1018108a-ceed-4fd7-8ad8-23f7478d7cae-home\") pod \"1018108a-ceed-4fd7-8ad8-23f7478d7cae\" (UID: \"1018108a-ceed-4fd7-8ad8-23f7478d7cae\") "
Nov 03 22:48:40.309880 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:40.309866    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1018108a-ceed-4fd7-8ad8-23f7478d7cae-kube-api-access-swhwv\" (UniqueName: \"kubernetes.io/projected/1018108a-ceed-4fd7-8ad8-23f7478d7cae-kube-api-access-swhwv\") pod \"1018108a-ceed-4fd7-8ad8-23f7478d7cae\" (UID: \"1018108a-ceed-4fd7-8ad8-23f7478d7cae\") "
Nov 03 22:48:40.309992 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:40.309976    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1018108a-ceed-4fd7-8ad8-23f7478d7cae-tmp-dir" pod "1018108a-ceed-4fd7-8ad8-23f7478d7cae" (UID: "1018108a-ceed-4fd7-8ad8-23f7478d7cae"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:48:40.310159 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:40.310140    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1018108a-ceed-4fd7-8ad8-23f7478d7cae-home" pod "1018108a-ceed-4fd7-8ad8-23f7478d7cae" (UID: "1018108a-ceed-4fd7-8ad8-23f7478d7cae"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:48:40.311091 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:40.311063    4431 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1018108a-ceed-4fd7-8ad8-23f7478d7cae-home\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:48:40.311303 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:40.311285    4431 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1018108a-ceed-4fd7-8ad8-23f7478d7cae-model-cache\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:48:40.311420 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:40.311409    4431 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1018108a-ceed-4fd7-8ad8-23f7478d7cae-tmp-dir\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:48:40.329358 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:40.313241    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1018108a-ceed-4fd7-8ad8-23f7478d7cae-dshm" pod "1018108a-ceed-4fd7-8ad8-23f7478d7cae" (UID: "1018108a-ceed-4fd7-8ad8-23f7478d7cae"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:48:40.329358 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:40.316034    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1018108a-ceed-4fd7-8ad8-23f7478d7cae-tls-certs" pod "1018108a-ceed-4fd7-8ad8-23f7478d7cae" (UID: "1018108a-ceed-4fd7-8ad8-23f7478d7cae"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:48:40.331017 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:40.330989    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1018108a-ceed-4fd7-8ad8-23f7478d7cae-kube-api-access-swhwv" pod "1018108a-ceed-4fd7-8ad8-23f7478d7cae" (UID: "1018108a-ceed-4fd7-8ad8-23f7478d7cae"). InnerVolumeSpecName "kube-api-access-swhwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:48:40.412984 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:40.412949    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swhwv\" (UniqueName: \"kubernetes.io/projected/1018108a-ceed-4fd7-8ad8-23f7478d7cae-kube-api-access-swhwv\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:48:40.412984 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:40.412980    4431 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1018108a-ceed-4fd7-8ad8-23f7478d7cae-tls-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:48:40.412984 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:40.412991    4431 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1018108a-ceed-4fd7-8ad8-23f7478d7cae-dshm\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:48:40.635177 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:40.635098    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-9e540d-79df886dc8-q6kwt" podUID="2ec1402b-a0e6-4dc7-acf4-0644c9be7d81" containerName="payload-processing" containerID="cri-o://1d19a42624a4d40b2d553b49b767267fdcc9ee4ed3ae7b861fc8786670bbeb81" gracePeriod=25
Nov 03 22:48:40.719823 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:40.719787    4431 generic.go:356] "Generic (PLEG): container finished" podID="1018108a-ceed-4fd7-8ad8-23f7478d7cae" containerID="f984a6dd41f37095f6790345dcbd010e24b408e84dac8664569844455323b1e0" exitCode=0
Nov 03 22:48:40.720010 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:40.719853    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-71a7df/test-model-71a7df-kserve-6967965559-nd2vz"
Nov 03 22:48:40.720010 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:40.719868    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-71a7df/test-model-71a7df-kserve-6967965559-nd2vz" event={"ID":"1018108a-ceed-4fd7-8ad8-23f7478d7cae","Type":"ContainerDied","Data":"f984a6dd41f37095f6790345dcbd010e24b408e84dac8664569844455323b1e0"}
Nov 03 22:48:40.720010 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:40.719917    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-71a7df/test-model-71a7df-kserve-6967965559-nd2vz" event={"ID":"1018108a-ceed-4fd7-8ad8-23f7478d7cae","Type":"ContainerDied","Data":"2c58d8eb8b137c119550ab65409c5e5dafe21dad022859c08e5a11282d556d1c"}
Nov 03 22:48:40.720010 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:40.719939    4431 scope.go:122] "RemoveContainer" containerID="f984a6dd41f37095f6790345dcbd010e24b408e84dac8664569844455323b1e0"
Nov 03 22:48:40.749188 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:40.749155    4431 scope.go:122] "RemoveContainer" containerID="f984a6dd41f37095f6790345dcbd010e24b408e84dac8664569844455323b1e0"
Nov 03 22:48:40.749576 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:48:40.749542    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f984a6dd41f37095f6790345dcbd010e24b408e84dac8664569844455323b1e0\": container with ID starting with f984a6dd41f37095f6790345dcbd010e24b408e84dac8664569844455323b1e0 not found: ID does not exist" containerID="f984a6dd41f37095f6790345dcbd010e24b408e84dac8664569844455323b1e0"
Nov 03 22:48:40.749683 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:40.749591    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f984a6dd41f37095f6790345dcbd010e24b408e84dac8664569844455323b1e0"} err="failed to get container status \"f984a6dd41f37095f6790345dcbd010e24b408e84dac8664569844455323b1e0\": rpc error: code = NotFound desc = could not find container \"f984a6dd41f37095f6790345dcbd010e24b408e84dac8664569844455323b1e0\": container with ID starting with f984a6dd41f37095f6790345dcbd010e24b408e84dac8664569844455323b1e0 not found: ID does not exist"
Nov 03 22:48:40.774332 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:40.774287    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-71a7df/test-model-71a7df-kserve-6967965559-nd2vz"]
Nov 03 22:48:40.778101 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:40.778071    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-71a7df/test-model-71a7df-kserve-6967965559-nd2vz"]
Nov 03 22:48:40.822542 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:40.822491    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:48:40.917267 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:40.917244    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9e540d-79df886dc8-q6kwt"
Nov 03 22:48:41.022546 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:41.022518    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ec1402b-a0e6-4dc7-acf4-0644c9be7d81-kube-api-access-b7j57\" (UniqueName: \"kubernetes.io/projected/2ec1402b-a0e6-4dc7-acf4-0644c9be7d81-kube-api-access-b7j57\") pod \"2ec1402b-a0e6-4dc7-acf4-0644c9be7d81\" (UID: \"2ec1402b-a0e6-4dc7-acf4-0644c9be7d81\") "
Nov 03 22:48:41.022702 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:41.022630    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ec1402b-a0e6-4dc7-acf4-0644c9be7d81-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ec1402b-a0e6-4dc7-acf4-0644c9be7d81-plugins-config-volume\") pod \"2ec1402b-a0e6-4dc7-acf4-0644c9be7d81\" (UID: \"2ec1402b-a0e6-4dc7-acf4-0644c9be7d81\") "
Nov 03 22:48:41.023009 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:41.022982    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ec1402b-a0e6-4dc7-acf4-0644c9be7d81-plugins-config-volume" pod "2ec1402b-a0e6-4dc7-acf4-0644c9be7d81" (UID: "2ec1402b-a0e6-4dc7-acf4-0644c9be7d81"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:48:41.023177 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:41.023160    4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ec1402b-a0e6-4dc7-acf4-0644c9be7d81-plugins-config-volume\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:48:41.024769 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:41.024747    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ec1402b-a0e6-4dc7-acf4-0644c9be7d81-kube-api-access-b7j57" pod "2ec1402b-a0e6-4dc7-acf4-0644c9be7d81" (UID: "2ec1402b-a0e6-4dc7-acf4-0644c9be7d81"). InnerVolumeSpecName "kube-api-access-b7j57". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:48:41.124353 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:41.124314    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7j57\" (UniqueName: \"kubernetes.io/projected/2ec1402b-a0e6-4dc7-acf4-0644c9be7d81-kube-api-access-b7j57\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:48:41.729505 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:41.729474    4431 generic.go:356] "Generic (PLEG): container finished" podID="2ec1402b-a0e6-4dc7-acf4-0644c9be7d81" containerID="1d19a42624a4d40b2d553b49b767267fdcc9ee4ed3ae7b861fc8786670bbeb81" exitCode=0
Nov 03 22:48:41.729960 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:41.729522    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9e540d-79df886dc8-q6kwt"
Nov 03 22:48:41.729960 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:41.729553    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9e540d-79df886dc8-q6kwt" event={"ID":"2ec1402b-a0e6-4dc7-acf4-0644c9be7d81","Type":"ContainerDied","Data":"1d19a42624a4d40b2d553b49b767267fdcc9ee4ed3ae7b861fc8786670bbeb81"}
Nov 03 22:48:41.729960 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:41.729612    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9e540d-79df886dc8-q6kwt" event={"ID":"2ec1402b-a0e6-4dc7-acf4-0644c9be7d81","Type":"ContainerDied","Data":"0cd3543ef7a9bf91983426ac42ada9b9fdfd9df6e4ce43d95ed770e73a997a79"}
Nov 03 22:48:41.729960 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:41.729643    4431 scope.go:122] "RemoveContainer" containerID="1d19a42624a4d40b2d553b49b767267fdcc9ee4ed3ae7b861fc8786670bbeb81"
Nov 03 22:48:41.740822 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:41.740805    4431 scope.go:122] "RemoveContainer" containerID="1d19a42624a4d40b2d553b49b767267fdcc9ee4ed3ae7b861fc8786670bbeb81"
Nov 03 22:48:41.741089 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:48:41.741072    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d19a42624a4d40b2d553b49b767267fdcc9ee4ed3ae7b861fc8786670bbeb81\": container with ID starting with 1d19a42624a4d40b2d553b49b767267fdcc9ee4ed3ae7b861fc8786670bbeb81 not found: ID does not exist" containerID="1d19a42624a4d40b2d553b49b767267fdcc9ee4ed3ae7b861fc8786670bbeb81"
Nov 03 22:48:41.741131 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:41.741099    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d19a42624a4d40b2d553b49b767267fdcc9ee4ed3ae7b861fc8786670bbeb81"} err="failed to get container status \"1d19a42624a4d40b2d553b49b767267fdcc9ee4ed3ae7b861fc8786670bbeb81\": rpc error: code = NotFound desc = could not find container \"1d19a42624a4d40b2d553b49b767267fdcc9ee4ed3ae7b861fc8786670bbeb81\": container with ID starting with 1d19a42624a4d40b2d553b49b767267fdcc9ee4ed3ae7b861fc8786670bbeb81 not found: ID does not exist"
Nov 03 22:48:41.785303 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:41.785275    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9e540d-79df886dc8-q6kwt"]
Nov 03 22:48:41.797555 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:41.797526    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9e540d-79df886dc8-q6kwt"]
Nov 03 22:48:42.089539 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:42.089461    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1018108a-ceed-4fd7-8ad8-23f7478d7cae" path="/var/lib/kubelet/pods/1018108a-ceed-4fd7-8ad8-23f7478d7cae/volumes"
Nov 03 22:48:42.089931 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:42.089917    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ec1402b-a0e6-4dc7-acf4-0644c9be7d81" path="/var/lib/kubelet/pods/2ec1402b-a0e6-4dc7-acf4-0644c9be7d81/volumes"
Nov 03 22:48:42.311128 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:42.311089    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:48:43.896837 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:43.896807    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:48:45.421645 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:45.421610    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:48:46.998592 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:46.998559    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:48:48.510239 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:48.510179    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:48:48.703825 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:48.703789    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:48:50.330196 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:50.330159    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:48:51.810424 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:51.810391    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:48:53.376834 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:53.376801    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:48:53.602149 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:53.602117    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:48:54.653875 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:54.653839    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:48:55.171228 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:55.171185    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:48:55.314321 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:55.314287    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:48:55.462696 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:55.462612    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:48:55.841034 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:55.840955    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:48:57.220320 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:57.220283    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:48:58.823591 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:48:58.823545    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:49:00.507917 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:00.507869    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:49:01.260862 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:49:01.260828    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:49:01.915306 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:01.915254    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:49:02.264094 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:02.263957    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:49:02.426119 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:02.426072    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:49:02.529653 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:02.529559    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-sf6vl"]
Nov 03 22:49:02.529944 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:02.529876    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-sf6vl" podUID="2c772f6e-d9c8-4dbe-aff9-b6b6026032ae" containerName="istio-proxy" containerID="cri-o://c8b0389c8d6b5eba8941413bf88b058c2e6b46b78ec8864e78f8edea0e504566" gracePeriod=30
Nov 03 22:49:02.620158 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:02.615400    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:49:02.770108 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:02.770075    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:49:02.929862 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:02.929827    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:49:03.219783 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:03.219696    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:49:04.319170 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:04.319135    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-281732/auth-test-model-281732-kserve-766d97949c-68cv5"]
Nov 03 22:49:04.320148 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:04.320125    4431 cpu_manager.go:414] "containerMap: removing container" podUID="2ec1402b-a0e6-4dc7-acf4-0644c9be7d81" containerName="payload-processing"
Nov 03 22:49:04.320148 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:04.320145    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ec1402b-a0e6-4dc7-acf4-0644c9be7d81" containerName="payload-processing"
Nov 03 22:49:04.320365 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:04.320167    4431 cpu_manager.go:414] "containerMap: removing container" podUID="152e2bae-faf4-4d6f-9791-01fbbe4581c3" containerName="discovery"
Nov 03 22:49:04.320365 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:04.320173    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="152e2bae-faf4-4d6f-9791-01fbbe4581c3" containerName="discovery"
Nov 03 22:49:04.320365 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:04.320204    4431 cpu_manager.go:414] "containerMap: removing container" podUID="1018108a-ceed-4fd7-8ad8-23f7478d7cae" containerName="main"
Nov 03 22:49:04.320365 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:04.320209    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1018108a-ceed-4fd7-8ad8-23f7478d7cae" containerName="main"
Nov 03 22:49:04.320365 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:04.320341    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="1018108a-ceed-4fd7-8ad8-23f7478d7cae" containerName="main"
Nov 03 22:49:04.320365 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:04.320354    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ec1402b-a0e6-4dc7-acf4-0644c9be7d81" containerName="payload-processing"
Nov 03 22:49:04.320365 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:04.320362    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="152e2bae-faf4-4d6f-9791-01fbbe4581c3" containerName="discovery"
Nov 03 22:49:04.325876 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:04.325849    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-281732/auth-test-model-281732-kserve-766d97949c-68cv5"
Nov 03 22:49:04.341482 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:04.341458    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-281732\"/\"openshift-service-ca.crt\""
Nov 03 22:49:04.343032 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:04.343001    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-281732/auth-test-model-281732-kserve-766d97949c-68cv5"]
Nov 03 22:49:04.349708 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:04.349688    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-281732\"/\"kube-root-ca.crt\""
Nov 03 22:49:04.353370 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:04.353322    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-281732\"/\"default-dockercfg-lxrdt\""
Nov 03 22:49:04.354046 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:04.354021    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-281732\"/\"auth-test-model-281732-kserve-self-signed-certs\""
Nov 03 22:49:04.402185 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:04.402151    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:49:04.486795 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:04.486757    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7248574d-773d-4837-899b-d5d62166da9d-tmp-dir\") pod \"auth-test-model-281732-kserve-766d97949c-68cv5\" (UID: \"7248574d-773d-4837-899b-d5d62166da9d\") " pod="ai-tenant-e2e-shared-a-w4-281732/auth-test-model-281732-kserve-766d97949c-68cv5"
Nov 03 22:49:04.487011 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:04.486909    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqzz6\" (UniqueName: \"kubernetes.io/projected/7248574d-773d-4837-899b-d5d62166da9d-kube-api-access-bqzz6\") pod \"auth-test-model-281732-kserve-766d97949c-68cv5\" (UID: \"7248574d-773d-4837-899b-d5d62166da9d\") " pod="ai-tenant-e2e-shared-a-w4-281732/auth-test-model-281732-kserve-766d97949c-68cv5"
Nov 03 22:49:04.487011 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:04.486972    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7248574d-773d-4837-899b-d5d62166da9d-model-cache\") pod \"auth-test-model-281732-kserve-766d97949c-68cv5\" (UID: \"7248574d-773d-4837-899b-d5d62166da9d\") " pod="ai-tenant-e2e-shared-a-w4-281732/auth-test-model-281732-kserve-766d97949c-68cv5"
Nov 03 22:49:04.487131 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:04.487062    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7248574d-773d-4837-899b-d5d62166da9d-home\") pod \"auth-test-model-281732-kserve-766d97949c-68cv5\" (UID: \"7248574d-773d-4837-899b-d5d62166da9d\") " pod="ai-tenant-e2e-shared-a-w4-281732/auth-test-model-281732-kserve-766d97949c-68cv5"
Nov 03 22:49:04.487131 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:04.487117    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7248574d-773d-4837-899b-d5d62166da9d-dshm\") pod \"auth-test-model-281732-kserve-766d97949c-68cv5\" (UID: \"7248574d-773d-4837-899b-d5d62166da9d\") " pod="ai-tenant-e2e-shared-a-w4-281732/auth-test-model-281732-kserve-766d97949c-68cv5"
Nov 03 22:49:04.487282 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:04.487201    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7248574d-773d-4837-899b-d5d62166da9d-tls-certs\") pod \"auth-test-model-281732-kserve-766d97949c-68cv5\" (UID: \"7248574d-773d-4837-899b-d5d62166da9d\") " pod="ai-tenant-e2e-shared-a-w4-281732/auth-test-model-281732-kserve-766d97949c-68cv5"
Nov 03 22:49:04.540991 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:04.540937    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:49:04.589403 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:04.589306    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7248574d-773d-4837-899b-d5d62166da9d-home\") pod \"auth-test-model-281732-kserve-766d97949c-68cv5\" (UID: \"7248574d-773d-4837-899b-d5d62166da9d\") " pod="ai-tenant-e2e-shared-a-w4-281732/auth-test-model-281732-kserve-766d97949c-68cv5"
Nov 03 22:49:04.589701 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:04.589678    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7248574d-773d-4837-899b-d5d62166da9d-dshm\") pod \"auth-test-model-281732-kserve-766d97949c-68cv5\" (UID: \"7248574d-773d-4837-899b-d5d62166da9d\") " pod="ai-tenant-e2e-shared-a-w4-281732/auth-test-model-281732-kserve-766d97949c-68cv5"
Nov 03 22:49:04.589902 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:04.589887    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7248574d-773d-4837-899b-d5d62166da9d-tls-certs\") pod \"auth-test-model-281732-kserve-766d97949c-68cv5\" (UID: \"7248574d-773d-4837-899b-d5d62166da9d\") " pod="ai-tenant-e2e-shared-a-w4-281732/auth-test-model-281732-kserve-766d97949c-68cv5"
Nov 03 22:49:04.590111 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:04.590096    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7248574d-773d-4837-899b-d5d62166da9d-tmp-dir\") pod \"auth-test-model-281732-kserve-766d97949c-68cv5\" (UID: \"7248574d-773d-4837-899b-d5d62166da9d\") " pod="ai-tenant-e2e-shared-a-w4-281732/auth-test-model-281732-kserve-766d97949c-68cv5"
Nov 03 22:49:04.590321 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:04.590303    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqzz6\" (UniqueName: \"kubernetes.io/projected/7248574d-773d-4837-899b-d5d62166da9d-kube-api-access-bqzz6\") pod \"auth-test-model-281732-kserve-766d97949c-68cv5\" (UID: \"7248574d-773d-4837-899b-d5d62166da9d\") " pod="ai-tenant-e2e-shared-a-w4-281732/auth-test-model-281732-kserve-766d97949c-68cv5"
Nov 03 22:49:04.590455 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:04.590439    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7248574d-773d-4837-899b-d5d62166da9d-model-cache\") pod \"auth-test-model-281732-kserve-766d97949c-68cv5\" (UID: \"7248574d-773d-4837-899b-d5d62166da9d\") " pod="ai-tenant-e2e-shared-a-w4-281732/auth-test-model-281732-kserve-766d97949c-68cv5"
Nov 03 22:49:04.590660 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:04.590491    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7248574d-773d-4837-899b-d5d62166da9d-home\") pod \"auth-test-model-281732-kserve-766d97949c-68cv5\" (UID: \"7248574d-773d-4837-899b-d5d62166da9d\") " pod="ai-tenant-e2e-shared-a-w4-281732/auth-test-model-281732-kserve-766d97949c-68cv5"
Nov 03 22:49:04.591059 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:04.591037    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7248574d-773d-4837-899b-d5d62166da9d-tmp-dir\") pod \"auth-test-model-281732-kserve-766d97949c-68cv5\" (UID: \"7248574d-773d-4837-899b-d5d62166da9d\") " pod="ai-tenant-e2e-shared-a-w4-281732/auth-test-model-281732-kserve-766d97949c-68cv5"
Nov 03 22:49:04.591459 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:04.591437    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7248574d-773d-4837-899b-d5d62166da9d-model-cache\") pod \"auth-test-model-281732-kserve-766d97949c-68cv5\" (UID: \"7248574d-773d-4837-899b-d5d62166da9d\") " pod="ai-tenant-e2e-shared-a-w4-281732/auth-test-model-281732-kserve-766d97949c-68cv5"
Nov 03 22:49:04.601493 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:04.601463    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7248574d-773d-4837-899b-d5d62166da9d-dshm\") pod \"auth-test-model-281732-kserve-766d97949c-68cv5\" (UID: \"7248574d-773d-4837-899b-d5d62166da9d\") " pod="ai-tenant-e2e-shared-a-w4-281732/auth-test-model-281732-kserve-766d97949c-68cv5"
Nov 03 22:49:04.611233 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:04.607867    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7248574d-773d-4837-899b-d5d62166da9d-tls-certs\") pod \"auth-test-model-281732-kserve-766d97949c-68cv5\" (UID: \"7248574d-773d-4837-899b-d5d62166da9d\") " pod="ai-tenant-e2e-shared-a-w4-281732/auth-test-model-281732-kserve-766d97949c-68cv5"
Nov 03 22:49:04.620007 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:04.618060    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqzz6\" (UniqueName: \"kubernetes.io/projected/7248574d-773d-4837-899b-d5d62166da9d-kube-api-access-bqzz6\") pod \"auth-test-model-281732-kserve-766d97949c-68cv5\" (UID: \"7248574d-773d-4837-899b-d5d62166da9d\") " pod="ai-tenant-e2e-shared-a-w4-281732/auth-test-model-281732-kserve-766d97949c-68cv5"
Nov 03 22:49:04.648337 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:04.648295    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-281732/auth-test-model-281732-kserve-766d97949c-68cv5"
Nov 03 22:49:04.694822 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:04.694791    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:49:04.987038 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:04.987006    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-281732/auth-test-model-281732-kserve-766d97949c-68cv5"]
Nov 03 22:49:04.988239 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:49:04.988194    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7248574d_773d_4837_899b_d5d62166da9d.slice/crio-11ca6e0e73482f6b489b58d6c84f2c50a8fe793707224b52552dc32c0a7a1c68 WatchSource:0}: Error finding container 11ca6e0e73482f6b489b58d6c84f2c50a8fe793707224b52552dc32c0a7a1c68: Status 404 returned error can't find the container with id 11ca6e0e73482f6b489b58d6c84f2c50a8fe793707224b52552dc32c0a7a1c68
Nov 03 22:49:05.154765 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:05.154736    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:49:05.546186 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:05.546149    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t"]
Nov 03 22:49:05.552002 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:05.551976    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t"
Nov 03 22:49:05.574464 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:05.572664    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t"]
Nov 03 22:49:05.578358 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:05.578326    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-4f9bb0-openshift-default-dockercfg-x9qpt\""
Nov 03 22:49:05.641110 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:05.641077    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:49:05.719691 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:05.719656    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe16bcab-2fb0-425d-961c-adc59648bace-istio-envoy\") pod \"e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t\" (UID: \"fe16bcab-2fb0-425d-961c-adc59648bace\") " pod="openshift-ingress/e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t"
Nov 03 22:49:05.719908 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:05.719785    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe16bcab-2fb0-425d-961c-adc59648bace-istiod-ca-cert\") pod \"e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t\" (UID: \"fe16bcab-2fb0-425d-961c-adc59648bace\") " pod="openshift-ingress/e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t"
Nov 03 22:49:05.719908 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:05.719866    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bs4n\" (UniqueName: \"kubernetes.io/projected/fe16bcab-2fb0-425d-961c-adc59648bace-kube-api-access-8bs4n\") pod \"e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t\" (UID: \"fe16bcab-2fb0-425d-961c-adc59648bace\") " pod="openshift-ingress/e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t"
Nov 03 22:49:05.719908 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:05.719896    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe16bcab-2fb0-425d-961c-adc59648bace-istio-data\") pod \"e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t\" (UID: \"fe16bcab-2fb0-425d-961c-adc59648bace\") " pod="openshift-ingress/e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t"
Nov 03 22:49:05.720153 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:05.720013    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe16bcab-2fb0-425d-961c-adc59648bace-credential-socket\") pod \"e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t\" (UID: \"fe16bcab-2fb0-425d-961c-adc59648bace\") " pod="openshift-ingress/e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t"
Nov 03 22:49:05.720153 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:05.720125    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe16bcab-2fb0-425d-961c-adc59648bace-istio-podinfo\") pod \"e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t\" (UID: \"fe16bcab-2fb0-425d-961c-adc59648bace\") " pod="openshift-ingress/e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t"
Nov 03 22:49:05.720289 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:05.720176    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe16bcab-2fb0-425d-961c-adc59648bace-workload-socket\") pod \"e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t\" (UID: \"fe16bcab-2fb0-425d-961c-adc59648bace\") " pod="openshift-ingress/e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t"
Nov 03 22:49:05.720289 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:05.720277    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe16bcab-2fb0-425d-961c-adc59648bace-workload-certs\") pod \"e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t\" (UID: \"fe16bcab-2fb0-425d-961c-adc59648bace\") " pod="openshift-ingress/e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t"
Nov 03 22:49:05.720406 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:05.720320    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe16bcab-2fb0-425d-961c-adc59648bace-istio-token\") pod \"e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t\" (UID: \"fe16bcab-2fb0-425d-961c-adc59648bace\") " pod="openshift-ingress/e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t"
Nov 03 22:49:05.800138 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:05.800032    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:49:05.831238 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:05.827433    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe16bcab-2fb0-425d-961c-adc59648bace-istio-data\") pod \"e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t\" (UID: \"fe16bcab-2fb0-425d-961c-adc59648bace\") " pod="openshift-ingress/e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t"
Nov 03 22:49:05.831238 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:05.827507    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe16bcab-2fb0-425d-961c-adc59648bace-credential-socket\") pod \"e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t\" (UID: \"fe16bcab-2fb0-425d-961c-adc59648bace\") " pod="openshift-ingress/e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t"
Nov 03 22:49:05.831238 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:05.827609    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe16bcab-2fb0-425d-961c-adc59648bace-istio-podinfo\") pod \"e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t\" (UID: \"fe16bcab-2fb0-425d-961c-adc59648bace\") " pod="openshift-ingress/e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t"
Nov 03 22:49:05.831238 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:05.827657    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe16bcab-2fb0-425d-961c-adc59648bace-workload-socket\") pod \"e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t\" (UID: \"fe16bcab-2fb0-425d-961c-adc59648bace\") " pod="openshift-ingress/e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t"
Nov 03 22:49:05.831238 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:05.827686    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe16bcab-2fb0-425d-961c-adc59648bace-workload-certs\") pod \"e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t\" (UID: \"fe16bcab-2fb0-425d-961c-adc59648bace\") " pod="openshift-ingress/e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t"
Nov 03 22:49:05.831238 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:05.827728    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe16bcab-2fb0-425d-961c-adc59648bace-istio-token\") pod \"e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t\" (UID: \"fe16bcab-2fb0-425d-961c-adc59648bace\") " pod="openshift-ingress/e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t"
Nov 03 22:49:05.831238 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:05.827846    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe16bcab-2fb0-425d-961c-adc59648bace-istio-envoy\") pod \"e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t\" (UID: \"fe16bcab-2fb0-425d-961c-adc59648bace\") " pod="openshift-ingress/e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t"
Nov 03 22:49:05.831238 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:05.827954    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe16bcab-2fb0-425d-961c-adc59648bace-istiod-ca-cert\") pod \"e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t\" (UID: \"fe16bcab-2fb0-425d-961c-adc59648bace\") " pod="openshift-ingress/e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t"
Nov 03 22:49:05.831238 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:05.828062    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bs4n\" (UniqueName: \"kubernetes.io/projected/fe16bcab-2fb0-425d-961c-adc59648bace-kube-api-access-8bs4n\") pod \"e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t\" (UID: \"fe16bcab-2fb0-425d-961c-adc59648bace\") " pod="openshift-ingress/e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t"
Nov 03 22:49:05.831238 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:05.829013    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe16bcab-2fb0-425d-961c-adc59648bace-istio-data\") pod \"e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t\" (UID: \"fe16bcab-2fb0-425d-961c-adc59648bace\") " pod="openshift-ingress/e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t"
Nov 03 22:49:05.831238 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:05.829315    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe16bcab-2fb0-425d-961c-adc59648bace-credential-socket\") pod \"e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t\" (UID: \"fe16bcab-2fb0-425d-961c-adc59648bace\") " pod="openshift-ingress/e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t"
Nov 03 22:49:05.831238 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:05.830302    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe16bcab-2fb0-425d-961c-adc59648bace-workload-socket\") pod \"e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t\" (UID: \"fe16bcab-2fb0-425d-961c-adc59648bace\") " pod="openshift-ingress/e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t"
Nov 03 22:49:05.831238 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:05.830555    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe16bcab-2fb0-425d-961c-adc59648bace-workload-certs\") pod \"e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t\" (UID: \"fe16bcab-2fb0-425d-961c-adc59648bace\") " pod="openshift-ingress/e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t"
Nov 03 22:49:05.837259 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:05.835636    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe16bcab-2fb0-425d-961c-adc59648bace-istiod-ca-cert\") pod \"e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t\" (UID: \"fe16bcab-2fb0-425d-961c-adc59648bace\") " pod="openshift-ingress/e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t"
Nov 03 22:49:05.838797 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:05.838768    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe16bcab-2fb0-425d-961c-adc59648bace-istio-podinfo\") pod \"e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t\" (UID: \"fe16bcab-2fb0-425d-961c-adc59648bace\") " pod="openshift-ingress/e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t"
Nov 03 22:49:05.857880 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:05.853553    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bs4n\" (UniqueName: \"kubernetes.io/projected/fe16bcab-2fb0-425d-961c-adc59648bace-kube-api-access-8bs4n\") pod \"e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t\" (UID: \"fe16bcab-2fb0-425d-961c-adc59648bace\") " pod="openshift-ingress/e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t"
Nov 03 22:49:05.857880 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:05.856529    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe16bcab-2fb0-425d-961c-adc59648bace-istio-token\") pod \"e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t\" (UID: \"fe16bcab-2fb0-425d-961c-adc59648bace\") " pod="openshift-ingress/e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t"
Nov 03 22:49:05.857880 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:05.857310    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe16bcab-2fb0-425d-961c-adc59648bace-istio-envoy\") pod \"e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t\" (UID: \"fe16bcab-2fb0-425d-961c-adc59648bace\") " pod="openshift-ingress/e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t"
Nov 03 22:49:05.873719 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:05.873490    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t"
Nov 03 22:49:05.931164 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:05.930396    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-281732/auth-test-model-281732-kserve-766d97949c-68cv5" event={"ID":"7248574d-773d-4837-899b-d5d62166da9d","Type":"ContainerStarted","Data":"cae888beef22ca52559cbe23e662d32dcc3ce67bcc02a9183d34e34e2231acfe"}
Nov 03 22:49:05.931164 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:05.930440    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-281732/auth-test-model-281732-kserve-766d97949c-68cv5" event={"ID":"7248574d-773d-4837-899b-d5d62166da9d","Type":"ContainerStarted","Data":"11ca6e0e73482f6b489b58d6c84f2c50a8fe793707224b52552dc32c0a7a1c68"}
Nov 03 22:49:05.972655 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:05.972608    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-281732/auth-test-model-281732-kserve-766d97949c-68cv5" podStartSLOduration=1.972592941 podStartE2EDuration="1.972592941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:49:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:49:05.972077837 +0000 UTC m=+2169.997933699" watchObservedRunningTime="2026-11-03 22:49:05.972592941 +0000 UTC m=+2169.998448803"
Nov 03 22:49:06.209151 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:49:06.209121    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe16bcab_2fb0_425d_961c_adc59648bace.slice/crio-92202e432711cef9fb152fc9acbdc700d8cf6d1924e852577b9e815d8e5005e5 WatchSource:0}: Error finding container 92202e432711cef9fb152fc9acbdc700d8cf6d1924e852577b9e815d8e5005e5: Status 404 returned error can't find the container with id 92202e432711cef9fb152fc9acbdc700d8cf6d1924e852577b9e815d8e5005e5
Nov 03 22:49:06.209815 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:06.209789    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t"]
Nov 03 22:49:06.212324 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:06.212290    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:49:06.212430 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:06.212357    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:49:06.212430 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:06.212388    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:49:06.241841 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:06.241801    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:49:06.649043 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:06.648963    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-281732/auth-test-model-281732-kserve-766d97949c-68cv5"
Nov 03 22:49:06.942067 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:06.942025    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t" event={"ID":"fe16bcab-2fb0-425d-961c-adc59648bace","Type":"ContainerStarted","Data":"e0814691e68fe0fa1842c60813e59af9d1da0e745e0a14e3b15866d7a228c8c1"}
Nov 03 22:49:06.942425 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:06.942109    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t" event={"ID":"fe16bcab-2fb0-425d-961c-adc59648bace","Type":"ContainerStarted","Data":"92202e432711cef9fb152fc9acbdc700d8cf6d1924e852577b9e815d8e5005e5"}
Nov 03 22:49:06.997414 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:06.997362    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t" podStartSLOduration=1.997346504 podStartE2EDuration="1.997346504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:49:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:49:06.992481826 +0000 UTC m=+2171.018337688" watchObservedRunningTime="2026-11-03 22:49:06.997346504 +0000 UTC m=+2171.023202365"
Nov 03 22:49:07.822355 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:07.822320    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:49:07.877204 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:07.876976    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t"
Nov 03 22:49:07.886494 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:07.886471    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t"
Nov 03 22:49:07.915691 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:07.915666    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-sf6vl"
Nov 03 22:49:07.950650 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:07.950618    4431 generic.go:356] "Generic (PLEG): container finished" podID="2c772f6e-d9c8-4dbe-aff9-b6b6026032ae" containerID="c8b0389c8d6b5eba8941413bf88b058c2e6b46b78ec8864e78f8edea0e504566" exitCode=0
Nov 03 22:49:07.950826 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:07.950671    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-sf6vl"
Nov 03 22:49:07.950826 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:07.950701    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-sf6vl" event={"ID":"2c772f6e-d9c8-4dbe-aff9-b6b6026032ae","Type":"ContainerDied","Data":"c8b0389c8d6b5eba8941413bf88b058c2e6b46b78ec8864e78f8edea0e504566"}
Nov 03 22:49:07.950826 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:07.950758    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-sf6vl" event={"ID":"2c772f6e-d9c8-4dbe-aff9-b6b6026032ae","Type":"ContainerDied","Data":"9dfd6453e69d288f1e5b6897a49af83e0dd55ccfa362e7ed83c1c2bd83a378ed"}
Nov 03 22:49:07.950826 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:07.950782    4431 scope.go:122] "RemoveContainer" containerID="c8b0389c8d6b5eba8941413bf88b058c2e6b46b78ec8864e78f8edea0e504566"
Nov 03 22:49:07.951163 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:07.951141    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t"
Nov 03 22:49:07.952007 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:07.951867    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t"
Nov 03 22:49:07.962407 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:07.962387    4431 scope.go:122] "RemoveContainer" containerID="c8b0389c8d6b5eba8941413bf88b058c2e6b46b78ec8864e78f8edea0e504566"
Nov 03 22:49:07.962653 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:49:07.962634    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8b0389c8d6b5eba8941413bf88b058c2e6b46b78ec8864e78f8edea0e504566\": container with ID starting with c8b0389c8d6b5eba8941413bf88b058c2e6b46b78ec8864e78f8edea0e504566 not found: ID does not exist" containerID="c8b0389c8d6b5eba8941413bf88b058c2e6b46b78ec8864e78f8edea0e504566"
Nov 03 22:49:07.962704 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:07.962661    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8b0389c8d6b5eba8941413bf88b058c2e6b46b78ec8864e78f8edea0e504566"} err="failed to get container status \"c8b0389c8d6b5eba8941413bf88b058c2e6b46b78ec8864e78f8edea0e504566\": rpc error: code = NotFound desc = could not find container \"c8b0389c8d6b5eba8941413bf88b058c2e6b46b78ec8864e78f8edea0e504566\": container with ID starting with c8b0389c8d6b5eba8941413bf88b058c2e6b46b78ec8864e78f8edea0e504566 not found: ID does not exist"
Nov 03 22:49:07.965126 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:07.965110    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c772f6e-d9c8-4dbe-aff9-b6b6026032ae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c772f6e-d9c8-4dbe-aff9-b6b6026032ae-credential-socket\") pod \"2c772f6e-d9c8-4dbe-aff9-b6b6026032ae\" (UID: \"2c772f6e-d9c8-4dbe-aff9-b6b6026032ae\") "
Nov 03 22:49:07.965187 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:07.965143    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c772f6e-d9c8-4dbe-aff9-b6b6026032ae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c772f6e-d9c8-4dbe-aff9-b6b6026032ae-istio-data\") pod \"2c772f6e-d9c8-4dbe-aff9-b6b6026032ae\" (UID: \"2c772f6e-d9c8-4dbe-aff9-b6b6026032ae\") "
Nov 03 22:49:07.965317 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:07.965301    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c772f6e-d9c8-4dbe-aff9-b6b6026032ae-istio-token\" (UniqueName: \"kubernetes.io/projected/2c772f6e-d9c8-4dbe-aff9-b6b6026032ae-istio-token\") pod \"2c772f6e-d9c8-4dbe-aff9-b6b6026032ae\" (UID: \"2c772f6e-d9c8-4dbe-aff9-b6b6026032ae\") "
Nov 03 22:49:07.965380 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:07.965343    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c772f6e-d9c8-4dbe-aff9-b6b6026032ae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c772f6e-d9c8-4dbe-aff9-b6b6026032ae-istio-envoy\") pod \"2c772f6e-d9c8-4dbe-aff9-b6b6026032ae\" (UID: \"2c772f6e-d9c8-4dbe-aff9-b6b6026032ae\") "
Nov 03 22:49:07.965443 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:07.965379    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2c772f6e-d9c8-4dbe-aff9-b6b6026032ae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c772f6e-d9c8-4dbe-aff9-b6b6026032ae-istio-podinfo\") pod \"2c772f6e-d9c8-4dbe-aff9-b6b6026032ae\" (UID: \"2c772f6e-d9c8-4dbe-aff9-b6b6026032ae\") "
Nov 03 22:49:07.965443 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:07.965409    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c772f6e-d9c8-4dbe-aff9-b6b6026032ae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c772f6e-d9c8-4dbe-aff9-b6b6026032ae-workload-socket\") pod \"2c772f6e-d9c8-4dbe-aff9-b6b6026032ae\" (UID: \"2c772f6e-d9c8-4dbe-aff9-b6b6026032ae\") "
Nov 03 22:49:07.965443 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:07.965428    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c772f6e-d9c8-4dbe-aff9-b6b6026032ae-istio-data" pod "2c772f6e-d9c8-4dbe-aff9-b6b6026032ae" (UID: "2c772f6e-d9c8-4dbe-aff9-b6b6026032ae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:49:07.965443 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:07.965439    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c772f6e-d9c8-4dbe-aff9-b6b6026032ae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c772f6e-d9c8-4dbe-aff9-b6b6026032ae-istiod-ca-cert\") pod \"2c772f6e-d9c8-4dbe-aff9-b6b6026032ae\" (UID: \"2c772f6e-d9c8-4dbe-aff9-b6b6026032ae\") "
Nov 03 22:49:07.965660 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:07.965419    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c772f6e-d9c8-4dbe-aff9-b6b6026032ae-credential-socket" pod "2c772f6e-d9c8-4dbe-aff9-b6b6026032ae" (UID: "2c772f6e-d9c8-4dbe-aff9-b6b6026032ae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:49:07.965660 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:07.965567    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c772f6e-d9c8-4dbe-aff9-b6b6026032ae-kube-api-access-79lq8\" (UniqueName: \"kubernetes.io/projected/2c772f6e-d9c8-4dbe-aff9-b6b6026032ae-kube-api-access-79lq8\") pod \"2c772f6e-d9c8-4dbe-aff9-b6b6026032ae\" (UID: \"2c772f6e-d9c8-4dbe-aff9-b6b6026032ae\") "
Nov 03 22:49:07.965660 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:07.965603    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c772f6e-d9c8-4dbe-aff9-b6b6026032ae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c772f6e-d9c8-4dbe-aff9-b6b6026032ae-workload-certs\") pod \"2c772f6e-d9c8-4dbe-aff9-b6b6026032ae\" (UID: \"2c772f6e-d9c8-4dbe-aff9-b6b6026032ae\") "
Nov 03 22:49:07.965829 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:07.965681    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c772f6e-d9c8-4dbe-aff9-b6b6026032ae-workload-socket" pod "2c772f6e-d9c8-4dbe-aff9-b6b6026032ae" (UID: "2c772f6e-d9c8-4dbe-aff9-b6b6026032ae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:49:07.965929 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:07.965892    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c772f6e-d9c8-4dbe-aff9-b6b6026032ae-workload-certs" pod "2c772f6e-d9c8-4dbe-aff9-b6b6026032ae" (UID: "2c772f6e-d9c8-4dbe-aff9-b6b6026032ae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:49:07.965929 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:07.965892    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c772f6e-d9c8-4dbe-aff9-b6b6026032ae-istiod-ca-cert" pod "2c772f6e-d9c8-4dbe-aff9-b6b6026032ae" (UID: "2c772f6e-d9c8-4dbe-aff9-b6b6026032ae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:49:07.966571 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:07.966381    4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c772f6e-d9c8-4dbe-aff9-b6b6026032ae-workload-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:49:07.966571 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:07.966409    4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c772f6e-d9c8-4dbe-aff9-b6b6026032ae-istiod-ca-cert\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:49:07.966571 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:07.966425    4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c772f6e-d9c8-4dbe-aff9-b6b6026032ae-workload-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:49:07.966571 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:07.966438    4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c772f6e-d9c8-4dbe-aff9-b6b6026032ae-credential-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:49:07.966571 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:07.966463    4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c772f6e-d9c8-4dbe-aff9-b6b6026032ae-istio-data\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:49:07.967777 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:07.967750    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c772f6e-d9c8-4dbe-aff9-b6b6026032ae-istio-token" pod "2c772f6e-d9c8-4dbe-aff9-b6b6026032ae" (UID: "2c772f6e-d9c8-4dbe-aff9-b6b6026032ae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:49:07.967955 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:07.967935    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c772f6e-d9c8-4dbe-aff9-b6b6026032ae-istio-envoy" pod "2c772f6e-d9c8-4dbe-aff9-b6b6026032ae" (UID: "2c772f6e-d9c8-4dbe-aff9-b6b6026032ae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:49:07.968023 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:07.968006    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2c772f6e-d9c8-4dbe-aff9-b6b6026032ae-istio-podinfo" pod "2c772f6e-d9c8-4dbe-aff9-b6b6026032ae" (UID: "2c772f6e-d9c8-4dbe-aff9-b6b6026032ae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 22:49:07.968158 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:07.968140    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c772f6e-d9c8-4dbe-aff9-b6b6026032ae-kube-api-access-79lq8" pod "2c772f6e-d9c8-4dbe-aff9-b6b6026032ae" (UID: "2c772f6e-d9c8-4dbe-aff9-b6b6026032ae"). InnerVolumeSpecName "kube-api-access-79lq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:49:08.070340 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:08.069598    4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c772f6e-d9c8-4dbe-aff9-b6b6026032ae-istio-token\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:49:08.071005 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:08.070985    4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c772f6e-d9c8-4dbe-aff9-b6b6026032ae-istio-envoy\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:49:08.071165 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:08.071150    4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c772f6e-d9c8-4dbe-aff9-b6b6026032ae-istio-podinfo\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:49:08.071285 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:08.071269    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79lq8\" (UniqueName: \"kubernetes.io/projected/2c772f6e-d9c8-4dbe-aff9-b6b6026032ae-kube-api-access-79lq8\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:49:08.293185 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:08.293145    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-sf6vl"]
Nov 03 22:49:08.296730 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:08.296700    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-0a7065-openshift-default-56696cf555-sf6vl"]
Nov 03 22:49:09.366895 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:09.366859    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:49:09.512782 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:09.512750    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:49:09.660349 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:09.660248    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:49:09.806199 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:09.806169    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:49:10.093539 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:10.093508    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c772f6e-d9c8-4dbe-aff9-b6b6026032ae" path="/var/lib/kubelet/pods/2c772f6e-d9c8-4dbe-aff9-b6b6026032ae/volumes"
Nov 03 22:49:10.162560 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:10.162526    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:49:10.998989 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:10.998955    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:49:11.076977 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:11.076942    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-4f9bb0-58c4b77cfb-cdgpk"]
Nov 03 22:49:11.077671 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:11.077651    4431 cpu_manager.go:414] "containerMap: removing container" podUID="2c772f6e-d9c8-4dbe-aff9-b6b6026032ae" containerName="istio-proxy"
Nov 03 22:49:11.077783 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:11.077672    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c772f6e-d9c8-4dbe-aff9-b6b6026032ae" containerName="istio-proxy"
Nov 03 22:49:11.077871 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:11.077859    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c772f6e-d9c8-4dbe-aff9-b6b6026032ae" containerName="istio-proxy"
Nov 03 22:49:11.083155 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:11.083133    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4f9bb0-58c4b77cfb-cdgpk"
Nov 03 22:49:11.100074 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:11.100053    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-4f9bb0-dockercfg-8jmwz\""
Nov 03 22:49:11.105850 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:11.105826    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-4f9bb0\""
Nov 03 22:49:11.112234 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:11.111667    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-4f9bb0-58c4b77cfb-cdgpk"]
Nov 03 22:49:11.135263 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:11.134422    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7n4h\" (UniqueName: \"kubernetes.io/projected/96145989-02fa-41e0-9a8a-fa2d78f47a25-kube-api-access-p7n4h\") pod \"payload-processing-e2e-shared-b-w2-4f9bb0-58c4b77cfb-cdgpk\" (UID: \"96145989-02fa-41e0-9a8a-fa2d78f47a25\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4f9bb0-58c4b77cfb-cdgpk"
Nov 03 22:49:11.142397 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:11.142364    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96145989-02fa-41e0-9a8a-fa2d78f47a25-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-4f9bb0-58c4b77cfb-cdgpk\" (UID: \"96145989-02fa-41e0-9a8a-fa2d78f47a25\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4f9bb0-58c4b77cfb-cdgpk"
Nov 03 22:49:11.152366 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:11.152327    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4f9bb0-6f8b69d999-g9rgk"]
Nov 03 22:49:11.156798 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:11.156769    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:49:11.159938 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:11.159916    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4f9bb0-6f8b69d999-g9rgk"
Nov 03 22:49:11.175843 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:11.175809    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4f9bb0-6f8b69d999-g9rgk"]
Nov 03 22:49:11.267714 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:11.267625    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9a82175-bff2-4fda-a4ff-756a962b51fe-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-4f9bb0-6f8b69d999-g9rgk\" (UID: \"b9a82175-bff2-4fda-a4ff-756a962b51fe\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4f9bb0-6f8b69d999-g9rgk"
Nov 03 22:49:11.268031 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:11.268008    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7n4h\" (UniqueName: \"kubernetes.io/projected/96145989-02fa-41e0-9a8a-fa2d78f47a25-kube-api-access-p7n4h\") pod \"payload-processing-e2e-shared-b-w2-4f9bb0-58c4b77cfb-cdgpk\" (UID: \"96145989-02fa-41e0-9a8a-fa2d78f47a25\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4f9bb0-58c4b77cfb-cdgpk"
Nov 03 22:49:11.268537 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:11.268513    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7x8f\" (UniqueName: \"kubernetes.io/projected/b9a82175-bff2-4fda-a4ff-756a962b51fe-kube-api-access-q7x8f\") pod \"payload-pre-processing-e2e-shared-b-w2-4f9bb0-6f8b69d999-g9rgk\" (UID: \"b9a82175-bff2-4fda-a4ff-756a962b51fe\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4f9bb0-6f8b69d999-g9rgk"
Nov 03 22:49:11.268974 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:11.268954    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96145989-02fa-41e0-9a8a-fa2d78f47a25-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-4f9bb0-58c4b77cfb-cdgpk\" (UID: \"96145989-02fa-41e0-9a8a-fa2d78f47a25\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4f9bb0-58c4b77cfb-cdgpk"
Nov 03 22:49:11.270154 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:11.269943    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96145989-02fa-41e0-9a8a-fa2d78f47a25-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-4f9bb0-58c4b77cfb-cdgpk\" (UID: \"96145989-02fa-41e0-9a8a-fa2d78f47a25\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4f9bb0-58c4b77cfb-cdgpk"
Nov 03 22:49:11.316910 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:11.316871    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7n4h\" (UniqueName: \"kubernetes.io/projected/96145989-02fa-41e0-9a8a-fa2d78f47a25-kube-api-access-p7n4h\") pod \"payload-processing-e2e-shared-b-w2-4f9bb0-58c4b77cfb-cdgpk\" (UID: \"96145989-02fa-41e0-9a8a-fa2d78f47a25\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4f9bb0-58c4b77cfb-cdgpk"
Nov 03 22:49:11.371654 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:11.371617    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9a82175-bff2-4fda-a4ff-756a962b51fe-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-4f9bb0-6f8b69d999-g9rgk\" (UID: \"b9a82175-bff2-4fda-a4ff-756a962b51fe\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4f9bb0-6f8b69d999-g9rgk"
Nov 03 22:49:11.371858 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:11.371702    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7x8f\" (UniqueName: \"kubernetes.io/projected/b9a82175-bff2-4fda-a4ff-756a962b51fe-kube-api-access-q7x8f\") pod \"payload-pre-processing-e2e-shared-b-w2-4f9bb0-6f8b69d999-g9rgk\" (UID: \"b9a82175-bff2-4fda-a4ff-756a962b51fe\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4f9bb0-6f8b69d999-g9rgk"
Nov 03 22:49:11.372344 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:11.372322    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9a82175-bff2-4fda-a4ff-756a962b51fe-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-4f9bb0-6f8b69d999-g9rgk\" (UID: \"b9a82175-bff2-4fda-a4ff-756a962b51fe\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4f9bb0-6f8b69d999-g9rgk"
Nov 03 22:49:11.395435 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:11.395391    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7x8f\" (UniqueName: \"kubernetes.io/projected/b9a82175-bff2-4fda-a4ff-756a962b51fe-kube-api-access-q7x8f\") pod \"payload-pre-processing-e2e-shared-b-w2-4f9bb0-6f8b69d999-g9rgk\" (UID: \"b9a82175-bff2-4fda-a4ff-756a962b51fe\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4f9bb0-6f8b69d999-g9rgk"
Nov 03 22:49:11.398036 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:11.398009    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4f9bb0-58c4b77cfb-cdgpk"
Nov 03 22:49:11.483801 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:11.483512    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4f9bb0-6f8b69d999-g9rgk"
Nov 03 22:49:11.652782 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:11.652749    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:49:11.709976 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:11.709946    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-4f9bb0-58c4b77cfb-cdgpk"]
Nov 03 22:49:11.712072 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:49:11.712044    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96145989_02fa_41e0_9a8a_fa2d78f47a25.slice/crio-ae8c5622125d70ff1308e1b3479b5d306fa1a2f2f07158b4b14600c75e10a92a WatchSource:0}: Error finding container ae8c5622125d70ff1308e1b3479b5d306fa1a2f2f07158b4b14600c75e10a92a: Status 404 returned error can't find the container with id ae8c5622125d70ff1308e1b3479b5d306fa1a2f2f07158b4b14600c75e10a92a
Nov 03 22:49:11.806007 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:49:11.805978    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9a82175_bff2_4fda_a4ff_756a962b51fe.slice/crio-0eb91ace58723490379bd3cc1d2ce191dfcfa4ed14f2f2b99cae4719531bd449 WatchSource:0}: Error finding container 0eb91ace58723490379bd3cc1d2ce191dfcfa4ed14f2f2b99cae4719531bd449: Status 404 returned error can't find the container with id 0eb91ace58723490379bd3cc1d2ce191dfcfa4ed14f2f2b99cae4719531bd449
Nov 03 22:49:11.806181 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:11.806156    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4f9bb0-6f8b69d999-g9rgk"]
Nov 03 22:49:11.984491 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:11.984458    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4f9bb0-6f8b69d999-g9rgk" event={"ID":"b9a82175-bff2-4fda-a4ff-756a962b51fe","Type":"ContainerStarted","Data":"a65f72026c07410cd34af55e9af19769a6779dae5ea7725de8cdb10523ff2941"}
Nov 03 22:49:11.984671 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:11.984509    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4f9bb0-6f8b69d999-g9rgk" event={"ID":"b9a82175-bff2-4fda-a4ff-756a962b51fe","Type":"ContainerStarted","Data":"0eb91ace58723490379bd3cc1d2ce191dfcfa4ed14f2f2b99cae4719531bd449"}
Nov 03 22:49:11.984671 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:11.984549    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4f9bb0-6f8b69d999-g9rgk"
Nov 03 22:49:11.986912 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:11.986884    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4f9bb0-58c4b77cfb-cdgpk" event={"ID":"96145989-02fa-41e0-9a8a-fa2d78f47a25","Type":"ContainerStarted","Data":"8a4906584911033305fd2650d829607358fd82f7d2cb94891eaa1babef49cbd0"}
Nov 03 22:49:11.987029 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:11.986944    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4f9bb0-58c4b77cfb-cdgpk"
Nov 03 22:49:11.987029 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:11.986969    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4f9bb0-58c4b77cfb-cdgpk" event={"ID":"96145989-02fa-41e0-9a8a-fa2d78f47a25","Type":"ContainerStarted","Data":"ae8c5622125d70ff1308e1b3479b5d306fa1a2f2f07158b4b14600c75e10a92a"}
Nov 03 22:49:12.029878 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:12.029830    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4f9bb0-6f8b69d999-g9rgk" podStartSLOduration=1.029813007 podStartE2EDuration="1.029813007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:49:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:49:12.028059451 +0000 UTC m=+2176.053915310" watchObservedRunningTime="2026-11-03 22:49:12.029813007 +0000 UTC m=+2176.055668869"
Nov 03 22:49:12.070919 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:12.070811    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4f9bb0-58c4b77cfb-cdgpk" podStartSLOduration=1.070792609 podStartE2EDuration="1.070792609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:49:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:49:12.067472516 +0000 UTC m=+2176.093328379" watchObservedRunningTime="2026-11-03 22:49:12.070792609 +0000 UTC m=+2176.096648474"
Nov 03 22:49:13.300115 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:13.300078    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:49:14.649547 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:14.649513    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-281732/auth-test-model-281732-kserve-766d97949c-68cv5"
Nov 03 22:49:14.662688 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:14.662657    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-281732/auth-test-model-281732-kserve-766d97949c-68cv5"
Nov 03 22:49:14.749208 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:14.749175    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:49:15.031848 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:15.031818    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-281732/auth-test-model-281732-kserve-766d97949c-68cv5"
Nov 03 22:49:16.339158 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:16.339100    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:49:17.846091 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:17.846056    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:49:19.340360 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:19.340317    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:49:21.011057 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:21.011012    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:49:21.523485 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:21.523450    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-71a7df-5f4b644b-zwb9k"]
Nov 03 22:49:21.553405 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:21.553372    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-71a7df-fdd88b4b7-sdbdj"]
Nov 03 22:49:21.744997 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:21.744185    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:49:21.989467 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:21.989436    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:49:22.709141 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:22.709109    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:49:22.997960 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:22.997883    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4f9bb0-58c4b77cfb-cdgpk"
Nov 03 22:49:22.997960 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:22.997959    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4f9bb0-6f8b69d999-g9rgk"
Nov 03 22:49:24.264469 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:24.264437    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:49:25.727895 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:25.727861    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:49:26.523947 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:26.523882    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-71a7df-5f4b644b-zwb9k" podUID="53276d80-69e4-482f-a285-370be912c076" containerName="payload-processing" containerID="cri-o://f70d1e6d72150391be4dfcb606abcc8a3eda8b5f078841db85309734fbbf1ce5" gracePeriod=25
Nov 03 22:49:26.531872 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:49:26.531850    4431 logging.go:55] [core] [Channel #3046 SubChannel #3047]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.107:50052", ServerName: "10.133.0.107:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.107:50052: connect: connection refused"
Nov 03 22:49:26.554088 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:26.554041    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-71a7df-fdd88b4b7-sdbdj" podUID="693e1c0a-7f7c-4a6b-af46-2aa6e1974da5" containerName="payload-pre-processing" containerID="cri-o://957f028e49907d406209661a4165b9692f36e6b6b3e7a90c580efd9181d7483d" gracePeriod=25
Nov 03 22:49:26.808639 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:26.808612    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-71a7df-5f4b644b-zwb9k"
Nov 03 22:49:26.923942 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:26.923905    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53276d80-69e4-482f-a285-370be912c076-kube-api-access-hcbht\" (UniqueName: \"kubernetes.io/projected/53276d80-69e4-482f-a285-370be912c076-kube-api-access-hcbht\") pod \"53276d80-69e4-482f-a285-370be912c076\" (UID: \"53276d80-69e4-482f-a285-370be912c076\") "
Nov 03 22:49:26.924272 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:26.924250    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53276d80-69e4-482f-a285-370be912c076-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53276d80-69e4-482f-a285-370be912c076-plugins-config-volume\") pod \"53276d80-69e4-482f-a285-370be912c076\" (UID: \"53276d80-69e4-482f-a285-370be912c076\") "
Nov 03 22:49:26.924585 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:26.924562    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53276d80-69e4-482f-a285-370be912c076-plugins-config-volume" pod "53276d80-69e4-482f-a285-370be912c076" (UID: "53276d80-69e4-482f-a285-370be912c076"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:49:26.924833 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:26.924819    4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53276d80-69e4-482f-a285-370be912c076-plugins-config-volume\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:49:26.926101 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:26.926078    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53276d80-69e4-482f-a285-370be912c076-kube-api-access-hcbht" pod "53276d80-69e4-482f-a285-370be912c076" (UID: "53276d80-69e4-482f-a285-370be912c076"). InnerVolumeSpecName "kube-api-access-hcbht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:49:27.025795 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:27.025717    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcbht\" (UniqueName: \"kubernetes.io/projected/53276d80-69e4-482f-a285-370be912c076-kube-api-access-hcbht\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:49:27.101129 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:27.101096    4431 generic.go:356] "Generic (PLEG): container finished" podID="53276d80-69e4-482f-a285-370be912c076" containerID="f70d1e6d72150391be4dfcb606abcc8a3eda8b5f078841db85309734fbbf1ce5" exitCode=0
Nov 03 22:49:27.101297 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:27.101133    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-71a7df-5f4b644b-zwb9k"
Nov 03 22:49:27.101297 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:27.101154    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-71a7df-5f4b644b-zwb9k" event={"ID":"53276d80-69e4-482f-a285-370be912c076","Type":"ContainerDied","Data":"f70d1e6d72150391be4dfcb606abcc8a3eda8b5f078841db85309734fbbf1ce5"}
Nov 03 22:49:27.101297 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:27.101190    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-71a7df-5f4b644b-zwb9k" event={"ID":"53276d80-69e4-482f-a285-370be912c076","Type":"ContainerDied","Data":"97d7439eca4bb511fcb8d8cb850046fd4883e874aa573ed96c8551e9536c8fd6"}
Nov 03 22:49:27.101297 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:27.101211    4431 scope.go:122] "RemoveContainer" containerID="f70d1e6d72150391be4dfcb606abcc8a3eda8b5f078841db85309734fbbf1ce5"
Nov 03 22:49:27.112692 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:27.112669    4431 scope.go:122] "RemoveContainer" containerID="f70d1e6d72150391be4dfcb606abcc8a3eda8b5f078841db85309734fbbf1ce5"
Nov 03 22:49:27.112996 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:49:27.112974    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f70d1e6d72150391be4dfcb606abcc8a3eda8b5f078841db85309734fbbf1ce5\": container with ID starting with f70d1e6d72150391be4dfcb606abcc8a3eda8b5f078841db85309734fbbf1ce5 not found: ID does not exist" containerID="f70d1e6d72150391be4dfcb606abcc8a3eda8b5f078841db85309734fbbf1ce5"
Nov 03 22:49:27.113072 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:27.113007    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f70d1e6d72150391be4dfcb606abcc8a3eda8b5f078841db85309734fbbf1ce5"} err="failed to get container status \"f70d1e6d72150391be4dfcb606abcc8a3eda8b5f078841db85309734fbbf1ce5\": rpc error: code = NotFound desc = could not find container \"f70d1e6d72150391be4dfcb606abcc8a3eda8b5f078841db85309734fbbf1ce5\": container with ID starting with f70d1e6d72150391be4dfcb606abcc8a3eda8b5f078841db85309734fbbf1ce5 not found: ID does not exist"
Nov 03 22:49:27.152784 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:27.152754    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-71a7df-5f4b644b-zwb9k"]
Nov 03 22:49:27.156670 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:27.156644    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-71a7df-5f4b644b-zwb9k"]
Nov 03 22:49:27.281930 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:27.281853    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:49:27.532645 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:27.532559    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-71a7df-5f4b644b-zwb9k" podUID="53276d80-69e4-482f-a285-370be912c076" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.107:50052\" within 1s: context deadline exceeded"
Nov 03 22:49:27.532799 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:49:27.532652    4431 logging.go:55] [core] [Channel #3046 SubChannel #3047]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.107:50052", ServerName: "10.133.0.107:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.107:50052: operation was canceled"
Nov 03 22:49:28.088877 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:28.088845    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53276d80-69e4-482f-a285-370be912c076" path="/var/lib/kubelet/pods/53276d80-69e4-482f-a285-370be912c076/volumes"
Nov 03 22:49:28.839473 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:28.839417    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:49:30.428268 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:30.428209    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:49:31.263333 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:49:31.263291    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:49:32.371299 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:32.371267    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:49:33.923428 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:33.923391    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:49:35.498908 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:35.498870    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:49:36.533385 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:36.533348    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-71a7df-fdd88b4b7-sdbdj" podUID="693e1c0a-7f7c-4a6b-af46-2aa6e1974da5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 22:49:36.999501 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:36.999467    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:49:38.593420 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:38.593381    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:49:40.210135 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:40.210027    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:49:41.819783 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:41.819753    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:49:43.444940 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:43.442443    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:49:45.129050 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:45.129012    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:49:45.318908 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:45.318874    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-04988c/auth-test-model-04988c-kserve-78755cd7fc-k6tdb"]
Nov 03 22:49:45.320045 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:45.320008    4431 cpu_manager.go:414] "containerMap: removing container" podUID="53276d80-69e4-482f-a285-370be912c076" containerName="payload-processing"
Nov 03 22:49:45.320045 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:45.320046    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53276d80-69e4-482f-a285-370be912c076" containerName="payload-processing"
Nov 03 22:49:45.320326 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:45.320255    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="53276d80-69e4-482f-a285-370be912c076" containerName="payload-processing"
Nov 03 22:49:45.326067 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:45.326043    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-04988c/auth-test-model-04988c-kserve-78755cd7fc-k6tdb"
Nov 03 22:49:45.345839 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:45.345811    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-04988c/auth-test-model-04988c-kserve-78755cd7fc-k6tdb"]
Nov 03 22:49:45.353459 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:45.353429    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-04988c\"/\"openshift-service-ca.crt\""
Nov 03 22:49:45.353644 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:45.353504    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-04988c\"/\"default-dockercfg-h5ms4\""
Nov 03 22:49:45.355420 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:45.355401    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-04988c\"/\"auth-test-model-04988c-kserve-self-signed-certs\""
Nov 03 22:49:45.355525 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:45.355511    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-04988c\"/\"kube-root-ca.crt\""
Nov 03 22:49:45.392507 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:45.392432    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bed3920d-932a-4e44-ae02-45aac5cbf633-dshm\") pod \"auth-test-model-04988c-kserve-78755cd7fc-k6tdb\" (UID: \"bed3920d-932a-4e44-ae02-45aac5cbf633\") " pod="ai-tenant-e2e-shared-b-w4-04988c/auth-test-model-04988c-kserve-78755cd7fc-k6tdb"
Nov 03 22:49:45.392658 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:45.392559    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bed3920d-932a-4e44-ae02-45aac5cbf633-tmp-dir\") pod \"auth-test-model-04988c-kserve-78755cd7fc-k6tdb\" (UID: \"bed3920d-932a-4e44-ae02-45aac5cbf633\") " pod="ai-tenant-e2e-shared-b-w4-04988c/auth-test-model-04988c-kserve-78755cd7fc-k6tdb"
Nov 03 22:49:45.392712 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:45.392688    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bed3920d-932a-4e44-ae02-45aac5cbf633-model-cache\") pod \"auth-test-model-04988c-kserve-78755cd7fc-k6tdb\" (UID: \"bed3920d-932a-4e44-ae02-45aac5cbf633\") " pod="ai-tenant-e2e-shared-b-w4-04988c/auth-test-model-04988c-kserve-78755cd7fc-k6tdb"
Nov 03 22:49:45.392822 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:45.392805    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bed3920d-932a-4e44-ae02-45aac5cbf633-tls-certs\") pod \"auth-test-model-04988c-kserve-78755cd7fc-k6tdb\" (UID: \"bed3920d-932a-4e44-ae02-45aac5cbf633\") " pod="ai-tenant-e2e-shared-b-w4-04988c/auth-test-model-04988c-kserve-78755cd7fc-k6tdb"
Nov 03 22:49:45.392986 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:45.392960    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bed3920d-932a-4e44-ae02-45aac5cbf633-home\") pod \"auth-test-model-04988c-kserve-78755cd7fc-k6tdb\" (UID: \"bed3920d-932a-4e44-ae02-45aac5cbf633\") " pod="ai-tenant-e2e-shared-b-w4-04988c/auth-test-model-04988c-kserve-78755cd7fc-k6tdb"
Nov 03 22:49:45.393189 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:45.393168    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztvsh\" (UniqueName: \"kubernetes.io/projected/bed3920d-932a-4e44-ae02-45aac5cbf633-kube-api-access-ztvsh\") pod \"auth-test-model-04988c-kserve-78755cd7fc-k6tdb\" (UID: \"bed3920d-932a-4e44-ae02-45aac5cbf633\") " pod="ai-tenant-e2e-shared-b-w4-04988c/auth-test-model-04988c-kserve-78755cd7fc-k6tdb"
Nov 03 22:49:45.434792 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:45.434755    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:49:45.495186 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:45.495145    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bed3920d-932a-4e44-ae02-45aac5cbf633-home\") pod \"auth-test-model-04988c-kserve-78755cd7fc-k6tdb\" (UID: \"bed3920d-932a-4e44-ae02-45aac5cbf633\") " pod="ai-tenant-e2e-shared-b-w4-04988c/auth-test-model-04988c-kserve-78755cd7fc-k6tdb"
Nov 03 22:49:45.495376 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:45.495347    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztvsh\" (UniqueName: \"kubernetes.io/projected/bed3920d-932a-4e44-ae02-45aac5cbf633-kube-api-access-ztvsh\") pod \"auth-test-model-04988c-kserve-78755cd7fc-k6tdb\" (UID: \"bed3920d-932a-4e44-ae02-45aac5cbf633\") " pod="ai-tenant-e2e-shared-b-w4-04988c/auth-test-model-04988c-kserve-78755cd7fc-k6tdb"
Nov 03 22:49:45.495441 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:45.495416    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bed3920d-932a-4e44-ae02-45aac5cbf633-dshm\") pod \"auth-test-model-04988c-kserve-78755cd7fc-k6tdb\" (UID: \"bed3920d-932a-4e44-ae02-45aac5cbf633\") " pod="ai-tenant-e2e-shared-b-w4-04988c/auth-test-model-04988c-kserve-78755cd7fc-k6tdb"
Nov 03 22:49:45.495501 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:45.495489    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bed3920d-932a-4e44-ae02-45aac5cbf633-tmp-dir\") pod \"auth-test-model-04988c-kserve-78755cd7fc-k6tdb\" (UID: \"bed3920d-932a-4e44-ae02-45aac5cbf633\") " pod="ai-tenant-e2e-shared-b-w4-04988c/auth-test-model-04988c-kserve-78755cd7fc-k6tdb"
Nov 03 22:49:45.495573 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:45.495558    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bed3920d-932a-4e44-ae02-45aac5cbf633-model-cache\") pod \"auth-test-model-04988c-kserve-78755cd7fc-k6tdb\" (UID: \"bed3920d-932a-4e44-ae02-45aac5cbf633\") " pod="ai-tenant-e2e-shared-b-w4-04988c/auth-test-model-04988c-kserve-78755cd7fc-k6tdb"
Nov 03 22:49:45.495640 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:45.495594    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bed3920d-932a-4e44-ae02-45aac5cbf633-home\") pod \"auth-test-model-04988c-kserve-78755cd7fc-k6tdb\" (UID: \"bed3920d-932a-4e44-ae02-45aac5cbf633\") " pod="ai-tenant-e2e-shared-b-w4-04988c/auth-test-model-04988c-kserve-78755cd7fc-k6tdb"
Nov 03 22:49:45.495640 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:45.495635    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bed3920d-932a-4e44-ae02-45aac5cbf633-tls-certs\") pod \"auth-test-model-04988c-kserve-78755cd7fc-k6tdb\" (UID: \"bed3920d-932a-4e44-ae02-45aac5cbf633\") " pod="ai-tenant-e2e-shared-b-w4-04988c/auth-test-model-04988c-kserve-78755cd7fc-k6tdb"
Nov 03 22:49:45.495885 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:45.495855    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bed3920d-932a-4e44-ae02-45aac5cbf633-model-cache\") pod \"auth-test-model-04988c-kserve-78755cd7fc-k6tdb\" (UID: \"bed3920d-932a-4e44-ae02-45aac5cbf633\") " pod="ai-tenant-e2e-shared-b-w4-04988c/auth-test-model-04988c-kserve-78755cd7fc-k6tdb"
Nov 03 22:49:45.496002 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:45.495940    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bed3920d-932a-4e44-ae02-45aac5cbf633-tmp-dir\") pod \"auth-test-model-04988c-kserve-78755cd7fc-k6tdb\" (UID: \"bed3920d-932a-4e44-ae02-45aac5cbf633\") " pod="ai-tenant-e2e-shared-b-w4-04988c/auth-test-model-04988c-kserve-78755cd7fc-k6tdb"
Nov 03 22:49:45.497922 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:45.497897    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bed3920d-932a-4e44-ae02-45aac5cbf633-dshm\") pod \"auth-test-model-04988c-kserve-78755cd7fc-k6tdb\" (UID: \"bed3920d-932a-4e44-ae02-45aac5cbf633\") " pod="ai-tenant-e2e-shared-b-w4-04988c/auth-test-model-04988c-kserve-78755cd7fc-k6tdb"
Nov 03 22:49:45.498056 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:45.498040    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bed3920d-932a-4e44-ae02-45aac5cbf633-tls-certs\") pod \"auth-test-model-04988c-kserve-78755cd7fc-k6tdb\" (UID: \"bed3920d-932a-4e44-ae02-45aac5cbf633\") " pod="ai-tenant-e2e-shared-b-w4-04988c/auth-test-model-04988c-kserve-78755cd7fc-k6tdb"
Nov 03 22:49:45.521072 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:45.521045    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztvsh\" (UniqueName: \"kubernetes.io/projected/bed3920d-932a-4e44-ae02-45aac5cbf633-kube-api-access-ztvsh\") pod \"auth-test-model-04988c-kserve-78755cd7fc-k6tdb\" (UID: \"bed3920d-932a-4e44-ae02-45aac5cbf633\") " pod="ai-tenant-e2e-shared-b-w4-04988c/auth-test-model-04988c-kserve-78755cd7fc-k6tdb"
Nov 03 22:49:45.644775 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:45.644677    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-04988c/auth-test-model-04988c-kserve-78755cd7fc-k6tdb"
Nov 03 22:49:46.006415 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:46.006375    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-04988c/auth-test-model-04988c-kserve-78755cd7fc-k6tdb"]
Nov 03 22:49:46.007660 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:49:46.007634    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbed3920d_932a_4e44_ae02_45aac5cbf633.slice/crio-08a8f0ef6ffc238cca819386abdc678d0d1ee05e6697978596678715e57d89db WatchSource:0}: Error finding container 08a8f0ef6ffc238cca819386abdc678d0d1ee05e6697978596678715e57d89db: Status 404 returned error can't find the container with id 08a8f0ef6ffc238cca819386abdc678d0d1ee05e6697978596678715e57d89db
Nov 03 22:49:46.241194 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:46.241156    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-04988c/auth-test-model-04988c-kserve-78755cd7fc-k6tdb" event={"ID":"bed3920d-932a-4e44-ae02-45aac5cbf633","Type":"ContainerStarted","Data":"1c4b71ff880ff43e9d864940993befb193baef22419c0224f8f787989ef3b6b8"}
Nov 03 22:49:46.241604 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:46.241205    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-04988c/auth-test-model-04988c-kserve-78755cd7fc-k6tdb" event={"ID":"bed3920d-932a-4e44-ae02-45aac5cbf633","Type":"ContainerStarted","Data":"08a8f0ef6ffc238cca819386abdc678d0d1ee05e6697978596678715e57d89db"}
Nov 03 22:49:46.291961 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:46.291857    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-04988c/auth-test-model-04988c-kserve-78755cd7fc-k6tdb" podStartSLOduration=1.291842129 podStartE2EDuration="1.291842129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:49:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:49:46.288887913 +0000 UTC m=+2210.314743776" watchObservedRunningTime="2026-11-03 22:49:46.291842129 +0000 UTC m=+2210.317697991"
Nov 03 22:49:46.532531 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:46.532491    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-71a7df-fdd88b4b7-sdbdj" podUID="693e1c0a-7f7c-4a6b-af46-2aa6e1974da5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 22:49:46.645039 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:46.645004    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-04988c/auth-test-model-04988c-kserve-78755cd7fc-k6tdb"
Nov 03 22:49:46.844086 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:46.844061    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-71a7df-fdd88b4b7-sdbdj"
Nov 03 22:49:46.928147 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:46.928107    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/693e1c0a-7f7c-4a6b-af46-2aa6e1974da5-kube-api-access-m7svv\" (UniqueName: \"kubernetes.io/projected/693e1c0a-7f7c-4a6b-af46-2aa6e1974da5-kube-api-access-m7svv\") pod \"693e1c0a-7f7c-4a6b-af46-2aa6e1974da5\" (UID: \"693e1c0a-7f7c-4a6b-af46-2aa6e1974da5\") "
Nov 03 22:49:46.928461 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:46.928435    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/693e1c0a-7f7c-4a6b-af46-2aa6e1974da5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/693e1c0a-7f7c-4a6b-af46-2aa6e1974da5-plugins-config-volume\") pod \"693e1c0a-7f7c-4a6b-af46-2aa6e1974da5\" (UID: \"693e1c0a-7f7c-4a6b-af46-2aa6e1974da5\") "
Nov 03 22:49:46.928835 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:46.928808    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/693e1c0a-7f7c-4a6b-af46-2aa6e1974da5-plugins-config-volume" pod "693e1c0a-7f7c-4a6b-af46-2aa6e1974da5" (UID: "693e1c0a-7f7c-4a6b-af46-2aa6e1974da5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:49:46.929134 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:46.929117    4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/693e1c0a-7f7c-4a6b-af46-2aa6e1974da5-plugins-config-volume\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:49:46.929895 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:46.929867    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:49:46.930665 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:46.930641    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/693e1c0a-7f7c-4a6b-af46-2aa6e1974da5-kube-api-access-m7svv" pod "693e1c0a-7f7c-4a6b-af46-2aa6e1974da5" (UID: "693e1c0a-7f7c-4a6b-af46-2aa6e1974da5"). InnerVolumeSpecName "kube-api-access-m7svv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:49:47.030307 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:47.030272    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7svv\" (UniqueName: \"kubernetes.io/projected/693e1c0a-7f7c-4a6b-af46-2aa6e1974da5-kube-api-access-m7svv\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:49:47.250606 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:47.250565    4431 generic.go:356] "Generic (PLEG): container finished" podID="693e1c0a-7f7c-4a6b-af46-2aa6e1974da5" containerID="957f028e49907d406209661a4165b9692f36e6b6b3e7a90c580efd9181d7483d" exitCode=0
Nov 03 22:49:47.251056 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:47.250645    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-71a7df-fdd88b4b7-sdbdj"
Nov 03 22:49:47.251056 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:47.250685    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-71a7df-fdd88b4b7-sdbdj" event={"ID":"693e1c0a-7f7c-4a6b-af46-2aa6e1974da5","Type":"ContainerDied","Data":"957f028e49907d406209661a4165b9692f36e6b6b3e7a90c580efd9181d7483d"}
Nov 03 22:49:47.251056 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:47.250737    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-71a7df-fdd88b4b7-sdbdj" event={"ID":"693e1c0a-7f7c-4a6b-af46-2aa6e1974da5","Type":"ContainerDied","Data":"9551b0e802f1fca81633f838ec02b4643a3b2bca223d75aacba51fb6ebd0014f"}
Nov 03 22:49:47.251056 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:47.250765    4431 scope.go:122] "RemoveContainer" containerID="957f028e49907d406209661a4165b9692f36e6b6b3e7a90c580efd9181d7483d"
Nov 03 22:49:47.262048 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:47.262026    4431 scope.go:122] "RemoveContainer" containerID="957f028e49907d406209661a4165b9692f36e6b6b3e7a90c580efd9181d7483d"
Nov 03 22:49:47.262393 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:49:47.262372    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"957f028e49907d406209661a4165b9692f36e6b6b3e7a90c580efd9181d7483d\": container with ID starting with 957f028e49907d406209661a4165b9692f36e6b6b3e7a90c580efd9181d7483d not found: ID does not exist" containerID="957f028e49907d406209661a4165b9692f36e6b6b3e7a90c580efd9181d7483d"
Nov 03 22:49:47.262464 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:47.262404    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"957f028e49907d406209661a4165b9692f36e6b6b3e7a90c580efd9181d7483d"} err="failed to get container status \"957f028e49907d406209661a4165b9692f36e6b6b3e7a90c580efd9181d7483d\": rpc error: code = NotFound desc = could not find container \"957f028e49907d406209661a4165b9692f36e6b6b3e7a90c580efd9181d7483d\": container with ID starting with 957f028e49907d406209661a4165b9692f36e6b6b3e7a90c580efd9181d7483d not found: ID does not exist"
Nov 03 22:49:47.302623 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:47.302593    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-71a7df-fdd88b4b7-sdbdj"]
Nov 03 22:49:47.309765 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:47.309710    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-71a7df-fdd88b4b7-sdbdj"]
Nov 03 22:49:48.089364 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:48.089332    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="693e1c0a-7f7c-4a6b-af46-2aa6e1974da5" path="/var/lib/kubelet/pods/693e1c0a-7f7c-4a6b-af46-2aa6e1974da5/volumes"
Nov 03 22:49:48.459347 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:48.459314    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:49:48.644741 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:48.644703    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:49:50.018757 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:50.018722    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:49:51.441692 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:51.441660    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f9bb0-7444566c76-2zdrh"]
Nov 03 22:49:51.442410 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:51.442390    4431 cpu_manager.go:414] "containerMap: removing container" podUID="693e1c0a-7f7c-4a6b-af46-2aa6e1974da5" containerName="payload-pre-processing"
Nov 03 22:49:51.442471 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:51.442411    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="693e1c0a-7f7c-4a6b-af46-2aa6e1974da5" containerName="payload-pre-processing"
Nov 03 22:49:51.442558 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:51.442549    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="693e1c0a-7f7c-4a6b-af46-2aa6e1974da5" containerName="payload-pre-processing"
Nov 03 22:49:51.447603 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:51.447584    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f9bb0-7444566c76-2zdrh"
Nov 03 22:49:51.466778 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:51.466748    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f9bb0-7444566c76-2zdrh"]
Nov 03 22:49:51.467948 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:51.467923    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-4f9bb0-serving-cert\""
Nov 03 22:49:51.557320 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:51.557287    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:49:51.592812 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:51.592780    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56a3f25c-1413-4374-b87e-223dadff1c76-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-4f9bb0-7444566c76-2zdrh\" (UID: \"56a3f25c-1413-4374-b87e-223dadff1c76\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f9bb0-7444566c76-2zdrh"
Nov 03 22:49:51.592978 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:51.592827    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56a3f25c-1413-4374-b87e-223dadff1c76-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-4f9bb0-7444566c76-2zdrh\" (UID: \"56a3f25c-1413-4374-b87e-223dadff1c76\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f9bb0-7444566c76-2zdrh"
Nov 03 22:49:51.593022 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:51.592994    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6k69\" (UniqueName: \"kubernetes.io/projected/56a3f25c-1413-4374-b87e-223dadff1c76-kube-api-access-w6k69\") pod \"maas-api-e2e-shared-b-w2-4f9bb0-7444566c76-2zdrh\" (UID: \"56a3f25c-1413-4374-b87e-223dadff1c76\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f9bb0-7444566c76-2zdrh"
Nov 03 22:49:51.593067 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:51.593051    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56a3f25c-1413-4374-b87e-223dadff1c76-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-4f9bb0-7444566c76-2zdrh\" (UID: \"56a3f25c-1413-4374-b87e-223dadff1c76\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f9bb0-7444566c76-2zdrh"
Nov 03 22:49:51.695806 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:51.695711    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6k69\" (UniqueName: \"kubernetes.io/projected/56a3f25c-1413-4374-b87e-223dadff1c76-kube-api-access-w6k69\") pod \"maas-api-e2e-shared-b-w2-4f9bb0-7444566c76-2zdrh\" (UID: \"56a3f25c-1413-4374-b87e-223dadff1c76\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f9bb0-7444566c76-2zdrh"
Nov 03 22:49:51.696531 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:51.696503    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56a3f25c-1413-4374-b87e-223dadff1c76-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-4f9bb0-7444566c76-2zdrh\" (UID: \"56a3f25c-1413-4374-b87e-223dadff1c76\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f9bb0-7444566c76-2zdrh"
Nov 03 22:49:51.696842 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:51.696814    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56a3f25c-1413-4374-b87e-223dadff1c76-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-4f9bb0-7444566c76-2zdrh\" (UID: \"56a3f25c-1413-4374-b87e-223dadff1c76\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f9bb0-7444566c76-2zdrh"
Nov 03 22:49:51.697006 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:51.696860    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56a3f25c-1413-4374-b87e-223dadff1c76-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-4f9bb0-7444566c76-2zdrh\" (UID: \"56a3f25c-1413-4374-b87e-223dadff1c76\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f9bb0-7444566c76-2zdrh"
Nov 03 22:49:51.697867 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:51.697841    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56a3f25c-1413-4374-b87e-223dadff1c76-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-4f9bb0-7444566c76-2zdrh\" (UID: \"56a3f25c-1413-4374-b87e-223dadff1c76\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f9bb0-7444566c76-2zdrh"
Nov 03 22:49:51.699711 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:51.699036    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:49:51.700987 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:51.700963    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56a3f25c-1413-4374-b87e-223dadff1c76-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-4f9bb0-7444566c76-2zdrh\" (UID: \"56a3f25c-1413-4374-b87e-223dadff1c76\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f9bb0-7444566c76-2zdrh"
Nov 03 22:49:51.702715 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:51.702682    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56a3f25c-1413-4374-b87e-223dadff1c76-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-4f9bb0-7444566c76-2zdrh\" (UID: \"56a3f25c-1413-4374-b87e-223dadff1c76\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f9bb0-7444566c76-2zdrh"
Nov 03 22:49:51.720301 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:51.720270    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6k69\" (UniqueName: \"kubernetes.io/projected/56a3f25c-1413-4374-b87e-223dadff1c76-kube-api-access-w6k69\") pod \"maas-api-e2e-shared-b-w2-4f9bb0-7444566c76-2zdrh\" (UID: \"56a3f25c-1413-4374-b87e-223dadff1c76\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f9bb0-7444566c76-2zdrh"
Nov 03 22:49:51.762999 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:51.762965    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f9bb0-7444566c76-2zdrh"
Nov 03 22:49:52.076865 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:52.076839    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f9bb0-7444566c76-2zdrh"]
Nov 03 22:49:52.077917 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:49:52.077892    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56a3f25c_1413_4374_b87e_223dadff1c76.slice/crio-516c5a66e66ddc8e1e6103913443302999c8bf37a6029e842051a34fca86bad1 WatchSource:0}: Error finding container 516c5a66e66ddc8e1e6103913443302999c8bf37a6029e842051a34fca86bad1: Status 404 returned error can't find the container with id 516c5a66e66ddc8e1e6103913443302999c8bf37a6029e842051a34fca86bad1
Nov 03 22:49:52.288047 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:52.287998    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f9bb0-7444566c76-2zdrh" event={"ID":"56a3f25c-1413-4374-b87e-223dadff1c76","Type":"ContainerStarted","Data":"516c5a66e66ddc8e1e6103913443302999c8bf37a6029e842051a34fca86bad1"}
Nov 03 22:49:52.791787 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:52.791750    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-71a7df-86bdf57769-s22zx"]
Nov 03 22:49:52.792329 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:52.791977    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-71a7df-86bdf57769-s22zx" podUID="ddf32aeb-0884-4e84-aef8-19fa55f251e3" containerName="maas-api" containerID="cri-o://7fd45994dd6ff1dac740e065a8c8b408882cb175c7781a83411b4624f080a5ea" gracePeriod=30
Nov 03 22:49:52.893339 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:52.892697    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:49:53.266887 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:53.264043    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-71a7df-86bdf57769-s22zx"
Nov 03 22:49:53.304418 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:53.304380    4431 generic.go:356] "Generic (PLEG): container finished" podID="ddf32aeb-0884-4e84-aef8-19fa55f251e3" containerID="7fd45994dd6ff1dac740e065a8c8b408882cb175c7781a83411b4624f080a5ea" exitCode=0
Nov 03 22:49:53.304734 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:53.304434    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-71a7df-86bdf57769-s22zx"
Nov 03 22:49:53.312655 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:53.304455    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-71a7df-86bdf57769-s22zx" event={"ID":"ddf32aeb-0884-4e84-aef8-19fa55f251e3","Type":"ContainerDied","Data":"7fd45994dd6ff1dac740e065a8c8b408882cb175c7781a83411b4624f080a5ea"}
Nov 03 22:49:53.313785 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:53.313766    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f9bb0-7444566c76-2zdrh"
Nov 03 22:49:53.313958 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:53.313940    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-71a7df-86bdf57769-s22zx" event={"ID":"ddf32aeb-0884-4e84-aef8-19fa55f251e3","Type":"ContainerDied","Data":"ffac2faf2a8e995d80af3cb5528267b72086dd677244f35559193fd3b49ac2ac"}
Nov 03 22:49:53.314065 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:53.314051    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f9bb0-7444566c76-2zdrh" event={"ID":"56a3f25c-1413-4374-b87e-223dadff1c76","Type":"ContainerStarted","Data":"2cd951f8d941ac6029213de7557d42d3cd8228c4d28e746fe27093783e9bbc29"}
Nov 03 22:49:53.314170 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:53.314158    4431 scope.go:122] "RemoveContainer" containerID="7fd45994dd6ff1dac740e065a8c8b408882cb175c7781a83411b4624f080a5ea"
Nov 03 22:49:53.348374 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:53.348338    4431 scope.go:122] "RemoveContainer" containerID="7fd45994dd6ff1dac740e065a8c8b408882cb175c7781a83411b4624f080a5ea"
Nov 03 22:49:53.354433 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:49:53.354384    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fd45994dd6ff1dac740e065a8c8b408882cb175c7781a83411b4624f080a5ea\": container with ID starting with 7fd45994dd6ff1dac740e065a8c8b408882cb175c7781a83411b4624f080a5ea not found: ID does not exist" containerID="7fd45994dd6ff1dac740e065a8c8b408882cb175c7781a83411b4624f080a5ea"
Nov 03 22:49:53.354588 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:53.354443    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fd45994dd6ff1dac740e065a8c8b408882cb175c7781a83411b4624f080a5ea"} err="failed to get container status \"7fd45994dd6ff1dac740e065a8c8b408882cb175c7781a83411b4624f080a5ea\": rpc error: code = NotFound desc = could not find container \"7fd45994dd6ff1dac740e065a8c8b408882cb175c7781a83411b4624f080a5ea\": container with ID starting with 7fd45994dd6ff1dac740e065a8c8b408882cb175c7781a83411b4624f080a5ea not found: ID does not exist"
Nov 03 22:49:53.357569 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:53.357538    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddf32aeb-0884-4e84-aef8-19fa55f251e3-kube-api-access-zpkfj\" (UniqueName: \"kubernetes.io/projected/ddf32aeb-0884-4e84-aef8-19fa55f251e3-kube-api-access-zpkfj\") pod \"ddf32aeb-0884-4e84-aef8-19fa55f251e3\" (UID: \"ddf32aeb-0884-4e84-aef8-19fa55f251e3\") "
Nov 03 22:49:53.357864 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:53.357842    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ddf32aeb-0884-4e84-aef8-19fa55f251e3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddf32aeb-0884-4e84-aef8-19fa55f251e3-metrics-certs\") pod \"ddf32aeb-0884-4e84-aef8-19fa55f251e3\" (UID: \"ddf32aeb-0884-4e84-aef8-19fa55f251e3\") "
Nov 03 22:49:53.358066 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:53.358046    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddf32aeb-0884-4e84-aef8-19fa55f251e3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddf32aeb-0884-4e84-aef8-19fa55f251e3-openshift-service-ca\") pod \"ddf32aeb-0884-4e84-aef8-19fa55f251e3\" (UID: \"ddf32aeb-0884-4e84-aef8-19fa55f251e3\") "
Nov 03 22:49:53.358817 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:53.358795    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ddf32aeb-0884-4e84-aef8-19fa55f251e3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ddf32aeb-0884-4e84-aef8-19fa55f251e3-maas-api-tls\") pod \"ddf32aeb-0884-4e84-aef8-19fa55f251e3\" (UID: \"ddf32aeb-0884-4e84-aef8-19fa55f251e3\") "
Nov 03 22:49:53.361770 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:53.361748    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddf32aeb-0884-4e84-aef8-19fa55f251e3-openshift-service-ca" pod "ddf32aeb-0884-4e84-aef8-19fa55f251e3" (UID: "ddf32aeb-0884-4e84-aef8-19fa55f251e3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:49:53.364261 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:53.364191    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddf32aeb-0884-4e84-aef8-19fa55f251e3-metrics-certs" pod "ddf32aeb-0884-4e84-aef8-19fa55f251e3" (UID: "ddf32aeb-0884-4e84-aef8-19fa55f251e3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:49:53.368120 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:53.367814    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddf32aeb-0884-4e84-aef8-19fa55f251e3-kube-api-access-zpkfj" pod "ddf32aeb-0884-4e84-aef8-19fa55f251e3" (UID: "ddf32aeb-0884-4e84-aef8-19fa55f251e3"). InnerVolumeSpecName "kube-api-access-zpkfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:49:53.369870 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:53.369837    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddf32aeb-0884-4e84-aef8-19fa55f251e3-maas-api-tls" pod "ddf32aeb-0884-4e84-aef8-19fa55f251e3" (UID: "ddf32aeb-0884-4e84-aef8-19fa55f251e3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:49:53.381113 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:53.378849    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f9bb0-7444566c76-2zdrh" podStartSLOduration=2.107877376 podStartE2EDuration="2.378829673s" totalImagesPullingTime="270.952297ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:49:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:49:53.378017706 +0000 UTC m=+2217.403873569" watchObservedRunningTime="2026-11-03 22:49:53.378829673 +0000 UTC m=+2217.404685536"
Nov 03 22:49:53.395061 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:53.395026    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:49:53.461175 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:53.461082    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpkfj\" (UniqueName: \"kubernetes.io/projected/ddf32aeb-0884-4e84-aef8-19fa55f251e3-kube-api-access-zpkfj\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:49:53.461175 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:53.461111    4431 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddf32aeb-0884-4e84-aef8-19fa55f251e3-metrics-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:49:53.461175 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:53.461120    4431 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddf32aeb-0884-4e84-aef8-19fa55f251e3-openshift-service-ca\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:49:53.461175 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:53.461129    4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ddf32aeb-0884-4e84-aef8-19fa55f251e3-maas-api-tls\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:49:53.536554 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:53.536518    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:49:53.700868 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:53.700717    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:49:53.704374 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:53.704334    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-71a7df-86bdf57769-s22zx"]
Nov 03 22:49:53.711171 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:53.711143    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-71a7df-86bdf57769-s22zx"]
Nov 03 22:49:53.842522 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:53.842438    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:49:54.047990 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:54.047959    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:49:54.089352 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:54.089320    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddf32aeb-0884-4e84-aef8-19fa55f251e3" path="/var/lib/kubelet/pods/ddf32aeb-0884-4e84-aef8-19fa55f251e3/volumes"
Nov 03 22:49:55.190000 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:55.189968    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:49:55.645870 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:55.645790    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-04988c/auth-test-model-04988c-kserve-78755cd7fc-k6tdb"
Nov 03 22:49:55.658790 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:55.658765    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-04988c/auth-test-model-04988c-kserve-78755cd7fc-k6tdb"
Nov 03 22:49:56.341203 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:56.341171    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-04988c/auth-test-model-04988c-kserve-78755cd7fc-k6tdb"
Nov 03 22:49:56.904030 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:56.903992    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:49:58.418838 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:58.418798    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:49:59.321509 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:49:59.321479    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f9bb0-7444566c76-2zdrh"
Nov 03 22:50:00.016172 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:00.016136    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:50:00.459423 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:00.459388    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:50:00.612958 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:00.612798    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:50:00.733298 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:00.732367    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv"]
Nov 03 22:50:00.733298 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:00.732677    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv" podUID="f8b1f74b-851b-41ed-a643-f01042642e59" containerName="istio-proxy" containerID="cri-o://654b952ec5cff6ad73e51965d40370e209d7e9eed9c39f510d8348cbe95ab7b2" gracePeriod=30
Nov 03 22:50:00.796438 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:00.796398    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:50:01.261450 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:50:01.261412    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:50:01.310617 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:01.310582    4431 patch_prober.go:29] interesting pod/e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 22:50:01.310809 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:01.310644    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv" podUID="f8b1f74b-851b-41ed-a643-f01042642e59" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 22:50:01.605662 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:01.605570    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:50:01.748427 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:01.748386    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:50:01.970204 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:01.970168    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9"]
Nov 03 22:50:01.971520 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:01.971492    4431 cpu_manager.go:414] "containerMap: removing container" podUID="ddf32aeb-0884-4e84-aef8-19fa55f251e3" containerName="maas-api"
Nov 03 22:50:01.971520 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:01.971519    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddf32aeb-0884-4e84-aef8-19fa55f251e3" containerName="maas-api"
Nov 03 22:50:01.971716 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:01.971704    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddf32aeb-0884-4e84-aef8-19fa55f251e3" containerName="maas-api"
Nov 03 22:50:01.977140 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:01.977114    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9"
Nov 03 22:50:02.004941 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:02.004909    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9"]
Nov 03 22:50:02.007578 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:02.007554    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-b023a2-openshift-default-dockercfg-fxgqz\""
Nov 03 22:50:02.081108 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:02.081064    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:50:02.136821 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:02.136782    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db62e992-8f6d-4cba-b260-02bf3c06e839-istiod-ca-cert\") pod \"e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9\" (UID: \"db62e992-8f6d-4cba-b260-02bf3c06e839\") " pod="openshift-ingress/e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9"
Nov 03 22:50:02.137049 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:02.136886    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db62e992-8f6d-4cba-b260-02bf3c06e839-credential-socket\") pod \"e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9\" (UID: \"db62e992-8f6d-4cba-b260-02bf3c06e839\") " pod="openshift-ingress/e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9"
Nov 03 22:50:02.137049 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:02.136927    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db62e992-8f6d-4cba-b260-02bf3c06e839-istio-envoy\") pod \"e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9\" (UID: \"db62e992-8f6d-4cba-b260-02bf3c06e839\") " pod="openshift-ingress/e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9"
Nov 03 22:50:02.137049 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:02.137038    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db62e992-8f6d-4cba-b260-02bf3c06e839-istio-podinfo\") pod \"e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9\" (UID: \"db62e992-8f6d-4cba-b260-02bf3c06e839\") " pod="openshift-ingress/e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9"
Nov 03 22:50:02.137255 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:02.137079    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db62e992-8f6d-4cba-b260-02bf3c06e839-workload-certs\") pod \"e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9\" (UID: \"db62e992-8f6d-4cba-b260-02bf3c06e839\") " pod="openshift-ingress/e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9"
Nov 03 22:50:02.137255 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:02.137113    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmb4g\" (UniqueName: \"kubernetes.io/projected/db62e992-8f6d-4cba-b260-02bf3c06e839-kube-api-access-jmb4g\") pod \"e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9\" (UID: \"db62e992-8f6d-4cba-b260-02bf3c06e839\") " pod="openshift-ingress/e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9"
Nov 03 22:50:02.137255 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:02.137174    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db62e992-8f6d-4cba-b260-02bf3c06e839-istio-token\") pod \"e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9\" (UID: \"db62e992-8f6d-4cba-b260-02bf3c06e839\") " pod="openshift-ingress/e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9"
Nov 03 22:50:02.137493 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:02.137459    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db62e992-8f6d-4cba-b260-02bf3c06e839-istio-data\") pod \"e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9\" (UID: \"db62e992-8f6d-4cba-b260-02bf3c06e839\") " pod="openshift-ingress/e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9"
Nov 03 22:50:02.137644 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:02.137516    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db62e992-8f6d-4cba-b260-02bf3c06e839-workload-socket\") pod \"e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9\" (UID: \"db62e992-8f6d-4cba-b260-02bf3c06e839\") " pod="openshift-ingress/e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9"
Nov 03 22:50:02.232298 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:02.230068    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:50:02.253286 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:02.253240    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db62e992-8f6d-4cba-b260-02bf3c06e839-istio-podinfo\") pod \"e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9\" (UID: \"db62e992-8f6d-4cba-b260-02bf3c06e839\") " pod="openshift-ingress/e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9"
Nov 03 22:50:02.254554 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:02.254529    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db62e992-8f6d-4cba-b260-02bf3c06e839-workload-certs\") pod \"e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9\" (UID: \"db62e992-8f6d-4cba-b260-02bf3c06e839\") " pod="openshift-ingress/e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9"
Nov 03 22:50:02.254803 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:02.254782    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmb4g\" (UniqueName: \"kubernetes.io/projected/db62e992-8f6d-4cba-b260-02bf3c06e839-kube-api-access-jmb4g\") pod \"e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9\" (UID: \"db62e992-8f6d-4cba-b260-02bf3c06e839\") " pod="openshift-ingress/e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9"
Nov 03 22:50:02.254978 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:02.254962    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db62e992-8f6d-4cba-b260-02bf3c06e839-istio-token\") pod \"e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9\" (UID: \"db62e992-8f6d-4cba-b260-02bf3c06e839\") " pod="openshift-ingress/e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9"
Nov 03 22:50:02.255227 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:02.255193    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db62e992-8f6d-4cba-b260-02bf3c06e839-istio-data\") pod \"e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9\" (UID: \"db62e992-8f6d-4cba-b260-02bf3c06e839\") " pod="openshift-ingress/e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9"
Nov 03 22:50:02.255372 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:02.255356    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db62e992-8f6d-4cba-b260-02bf3c06e839-workload-socket\") pod \"e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9\" (UID: \"db62e992-8f6d-4cba-b260-02bf3c06e839\") " pod="openshift-ingress/e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9"
Nov 03 22:50:02.255532 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:02.255518    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db62e992-8f6d-4cba-b260-02bf3c06e839-istiod-ca-cert\") pod \"e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9\" (UID: \"db62e992-8f6d-4cba-b260-02bf3c06e839\") " pod="openshift-ingress/e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9"
Nov 03 22:50:02.255725 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:02.255711    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db62e992-8f6d-4cba-b260-02bf3c06e839-credential-socket\") pod \"e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9\" (UID: \"db62e992-8f6d-4cba-b260-02bf3c06e839\") " pod="openshift-ingress/e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9"
Nov 03 22:50:02.277089 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:02.269315    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db62e992-8f6d-4cba-b260-02bf3c06e839-istio-envoy\") pod \"e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9\" (UID: \"db62e992-8f6d-4cba-b260-02bf3c06e839\") " pod="openshift-ingress/e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9"
Nov 03 22:50:02.277089 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:02.257980    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db62e992-8f6d-4cba-b260-02bf3c06e839-istio-podinfo\") pod \"e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9\" (UID: \"db62e992-8f6d-4cba-b260-02bf3c06e839\") " pod="openshift-ingress/e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9"
Nov 03 22:50:02.277089 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:02.258515    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db62e992-8f6d-4cba-b260-02bf3c06e839-workload-socket\") pod \"e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9\" (UID: \"db62e992-8f6d-4cba-b260-02bf3c06e839\") " pod="openshift-ingress/e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9"
Nov 03 22:50:02.277089 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:02.258950    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db62e992-8f6d-4cba-b260-02bf3c06e839-credential-socket\") pod \"e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9\" (UID: \"db62e992-8f6d-4cba-b260-02bf3c06e839\") " pod="openshift-ingress/e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9"
Nov 03 22:50:02.277089 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:02.259037    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db62e992-8f6d-4cba-b260-02bf3c06e839-istiod-ca-cert\") pod \"e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9\" (UID: \"db62e992-8f6d-4cba-b260-02bf3c06e839\") " pod="openshift-ingress/e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9"
Nov 03 22:50:02.277089 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:02.259295    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db62e992-8f6d-4cba-b260-02bf3c06e839-istio-data\") pod \"e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9\" (UID: \"db62e992-8f6d-4cba-b260-02bf3c06e839\") " pod="openshift-ingress/e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9"
Nov 03 22:50:02.277089 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:02.257505    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db62e992-8f6d-4cba-b260-02bf3c06e839-workload-certs\") pod \"e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9\" (UID: \"db62e992-8f6d-4cba-b260-02bf3c06e839\") " pod="openshift-ingress/e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9"
Nov 03 22:50:02.277089 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:02.276163    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db62e992-8f6d-4cba-b260-02bf3c06e839-istio-envoy\") pod \"e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9\" (UID: \"db62e992-8f6d-4cba-b260-02bf3c06e839\") " pod="openshift-ingress/e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9"
Nov 03 22:50:02.292881 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:02.292848    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db62e992-8f6d-4cba-b260-02bf3c06e839-istio-token\") pod \"e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9\" (UID: \"db62e992-8f6d-4cba-b260-02bf3c06e839\") " pod="openshift-ingress/e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9"
Nov 03 22:50:02.294642 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:02.294610    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmb4g\" (UniqueName: \"kubernetes.io/projected/db62e992-8f6d-4cba-b260-02bf3c06e839-kube-api-access-jmb4g\") pod \"e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9\" (UID: \"db62e992-8f6d-4cba-b260-02bf3c06e839\") " pod="openshift-ingress/e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9"
Nov 03 22:50:02.594533 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:02.594439    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9"
Nov 03 22:50:02.937279 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:02.937210    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9"]
Nov 03 22:50:02.946443 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:50:02.946411    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb62e992_8f6d_4cba_b260_02bf3c06e839.slice/crio-f04b672d9c755d2ef75b868eb4b8ccfa3fd2842ff85502c39fdd50aa63e02207 WatchSource:0}: Error finding container f04b672d9c755d2ef75b868eb4b8ccfa3fd2842ff85502c39fdd50aa63e02207: Status 404 returned error can't find the container with id f04b672d9c755d2ef75b868eb4b8ccfa3fd2842ff85502c39fdd50aa63e02207
Nov 03 22:50:02.950060 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:02.950022    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:50:02.950173 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:02.950105    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:50:02.950173 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:02.950134    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:50:03.285044 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:03.284921    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:50:03.384589 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:03.384549    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9" event={"ID":"db62e992-8f6d-4cba-b260-02bf3c06e839","Type":"ContainerStarted","Data":"86de258efe3b36a42c2aacc19c5e4e31ecf699a43467b762c39378d16a036b58"}
Nov 03 22:50:03.384767 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:03.384620    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9" event={"ID":"db62e992-8f6d-4cba-b260-02bf3c06e839","Type":"ContainerStarted","Data":"f04b672d9c755d2ef75b868eb4b8ccfa3fd2842ff85502c39fdd50aa63e02207"}
Nov 03 22:50:03.437029 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:03.436939    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9" podStartSLOduration=2.436890057 podStartE2EDuration="2.436890057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:50:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:50:03.428320048 +0000 UTC m=+2227.454175909" watchObservedRunningTime="2026-11-03 22:50:03.436890057 +0000 UTC m=+2227.462745922"
Nov 03 22:50:03.594926 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:03.594842    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9"
Nov 03 22:50:04.601081 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:04.601049    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9"
Nov 03 22:50:04.929175 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:04.929137    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:50:05.398239 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:05.398139    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9"
Nov 03 22:50:05.399031 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:05.399004    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9"
Nov 03 22:50:05.888400 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:05.888358    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:50:06.011290 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:06.011228    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv"
Nov 03 22:50:06.051357 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:06.051313    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:50:06.127648 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:06.127610    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8b1f74b-851b-41ed-a643-f01042642e59-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8b1f74b-851b-41ed-a643-f01042642e59-istio-data\") pod \"f8b1f74b-851b-41ed-a643-f01042642e59\" (UID: \"f8b1f74b-851b-41ed-a643-f01042642e59\") "
Nov 03 22:50:06.127831 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:06.127768    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8b1f74b-851b-41ed-a643-f01042642e59-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8b1f74b-851b-41ed-a643-f01042642e59-istio-envoy\") pod \"f8b1f74b-851b-41ed-a643-f01042642e59\" (UID: \"f8b1f74b-851b-41ed-a643-f01042642e59\") "
Nov 03 22:50:06.127891 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:06.127829    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8b1f74b-851b-41ed-a643-f01042642e59-istio-token\" (UniqueName: \"kubernetes.io/projected/f8b1f74b-851b-41ed-a643-f01042642e59-istio-token\") pod \"f8b1f74b-851b-41ed-a643-f01042642e59\" (UID: \"f8b1f74b-851b-41ed-a643-f01042642e59\") "
Nov 03 22:50:06.127948 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:06.127930    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8b1f74b-851b-41ed-a643-f01042642e59-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8b1f74b-851b-41ed-a643-f01042642e59-workload-socket\") pod \"f8b1f74b-851b-41ed-a643-f01042642e59\" (UID: \"f8b1f74b-851b-41ed-a643-f01042642e59\") "
Nov 03 22:50:06.128094 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:06.128075    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f8b1f74b-851b-41ed-a643-f01042642e59-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8b1f74b-851b-41ed-a643-f01042642e59-istio-podinfo\") pod \"f8b1f74b-851b-41ed-a643-f01042642e59\" (UID: \"f8b1f74b-851b-41ed-a643-f01042642e59\") "
Nov 03 22:50:06.128151 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:06.128125    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8b1f74b-851b-41ed-a643-f01042642e59-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8b1f74b-851b-41ed-a643-f01042642e59-workload-certs\") pod \"f8b1f74b-851b-41ed-a643-f01042642e59\" (UID: \"f8b1f74b-851b-41ed-a643-f01042642e59\") "
Nov 03 22:50:06.128205 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:06.128164    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8b1f74b-851b-41ed-a643-f01042642e59-kube-api-access-w7plt\" (UniqueName: \"kubernetes.io/projected/f8b1f74b-851b-41ed-a643-f01042642e59-kube-api-access-w7plt\") pod \"f8b1f74b-851b-41ed-a643-f01042642e59\" (UID: \"f8b1f74b-851b-41ed-a643-f01042642e59\") "
Nov 03 22:50:06.128316 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:06.128201    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8b1f74b-851b-41ed-a643-f01042642e59-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8b1f74b-851b-41ed-a643-f01042642e59-credential-socket\") pod \"f8b1f74b-851b-41ed-a643-f01042642e59\" (UID: \"f8b1f74b-851b-41ed-a643-f01042642e59\") "
Nov 03 22:50:06.128316 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:06.128247    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8b1f74b-851b-41ed-a643-f01042642e59-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8b1f74b-851b-41ed-a643-f01042642e59-istiod-ca-cert\") pod \"f8b1f74b-851b-41ed-a643-f01042642e59\" (UID: \"f8b1f74b-851b-41ed-a643-f01042642e59\") "
Nov 03 22:50:06.129724 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:06.129313    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8b1f74b-851b-41ed-a643-f01042642e59-istiod-ca-cert" pod "f8b1f74b-851b-41ed-a643-f01042642e59" (UID: "f8b1f74b-851b-41ed-a643-f01042642e59"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:50:06.131535 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:06.131508    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8b1f74b-851b-41ed-a643-f01042642e59-istio-envoy" pod "f8b1f74b-851b-41ed-a643-f01042642e59" (UID: "f8b1f74b-851b-41ed-a643-f01042642e59"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:50:06.131747 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:06.131725    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8b1f74b-851b-41ed-a643-f01042642e59-workload-certs" pod "f8b1f74b-851b-41ed-a643-f01042642e59" (UID: "f8b1f74b-851b-41ed-a643-f01042642e59"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:50:06.132811 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:06.132784    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8b1f74b-851b-41ed-a643-f01042642e59-credential-socket" pod "f8b1f74b-851b-41ed-a643-f01042642e59" (UID: "f8b1f74b-851b-41ed-a643-f01042642e59"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:50:06.133483 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:06.133334    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8b1f74b-851b-41ed-a643-f01042642e59-workload-socket" pod "f8b1f74b-851b-41ed-a643-f01042642e59" (UID: "f8b1f74b-851b-41ed-a643-f01042642e59"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:50:06.134452 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:06.133724    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f8b1f74b-851b-41ed-a643-f01042642e59-istio-podinfo" pod "f8b1f74b-851b-41ed-a643-f01042642e59" (UID: "f8b1f74b-851b-41ed-a643-f01042642e59"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 22:50:06.134452 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:06.133884    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8b1f74b-851b-41ed-a643-f01042642e59-istio-token" pod "f8b1f74b-851b-41ed-a643-f01042642e59" (UID: "f8b1f74b-851b-41ed-a643-f01042642e59"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:50:06.134790 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:06.134766    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8b1f74b-851b-41ed-a643-f01042642e59-kube-api-access-w7plt" pod "f8b1f74b-851b-41ed-a643-f01042642e59" (UID: "f8b1f74b-851b-41ed-a643-f01042642e59"). InnerVolumeSpecName "kube-api-access-w7plt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:50:06.135324 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:06.135300    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8b1f74b-851b-41ed-a643-f01042642e59-istio-data" pod "f8b1f74b-851b-41ed-a643-f01042642e59" (UID: "f8b1f74b-851b-41ed-a643-f01042642e59"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:50:06.230258 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:06.230225    4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8b1f74b-851b-41ed-a643-f01042642e59-istio-envoy\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:50:06.230258 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:06.230255    4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8b1f74b-851b-41ed-a643-f01042642e59-istio-token\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:50:06.230258 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:06.230267    4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8b1f74b-851b-41ed-a643-f01042642e59-workload-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:50:06.230500 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:06.230276    4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8b1f74b-851b-41ed-a643-f01042642e59-istio-podinfo\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:50:06.230500 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:06.230285    4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8b1f74b-851b-41ed-a643-f01042642e59-workload-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:50:06.230500 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:06.230293    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7plt\" (UniqueName: \"kubernetes.io/projected/f8b1f74b-851b-41ed-a643-f01042642e59-kube-api-access-w7plt\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:50:06.230500 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:06.230302    4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8b1f74b-851b-41ed-a643-f01042642e59-credential-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:50:06.230500 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:06.230310    4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8b1f74b-851b-41ed-a643-f01042642e59-istiod-ca-cert\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:50:06.230500 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:06.230319    4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8b1f74b-851b-41ed-a643-f01042642e59-istio-data\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:50:06.407504 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:06.407409    4431 generic.go:356] "Generic (PLEG): container finished" podID="f8b1f74b-851b-41ed-a643-f01042642e59" containerID="654b952ec5cff6ad73e51965d40370e209d7e9eed9c39f510d8348cbe95ab7b2" exitCode=0
Nov 03 22:50:06.407504 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:06.407464    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv"
Nov 03 22:50:06.407715 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:06.407504    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv" event={"ID":"f8b1f74b-851b-41ed-a643-f01042642e59","Type":"ContainerDied","Data":"654b952ec5cff6ad73e51965d40370e209d7e9eed9c39f510d8348cbe95ab7b2"}
Nov 03 22:50:06.407715 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:06.407565    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv" event={"ID":"f8b1f74b-851b-41ed-a643-f01042642e59","Type":"ContainerDied","Data":"c3d5366be22c353713cb7dc95c4988637d5b891d68fa5d025569ad3ed4d56275"}
Nov 03 22:50:06.407715 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:06.407588    4431 scope.go:122] "RemoveContainer" containerID="654b952ec5cff6ad73e51965d40370e209d7e9eed9c39f510d8348cbe95ab7b2"
Nov 03 22:50:06.419933 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:06.419914    4431 scope.go:122] "RemoveContainer" containerID="654b952ec5cff6ad73e51965d40370e209d7e9eed9c39f510d8348cbe95ab7b2"
Nov 03 22:50:06.420223 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:50:06.420189    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"654b952ec5cff6ad73e51965d40370e209d7e9eed9c39f510d8348cbe95ab7b2\": container with ID starting with 654b952ec5cff6ad73e51965d40370e209d7e9eed9c39f510d8348cbe95ab7b2 not found: ID does not exist" containerID="654b952ec5cff6ad73e51965d40370e209d7e9eed9c39f510d8348cbe95ab7b2"
Nov 03 22:50:06.420298 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:06.420229    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"654b952ec5cff6ad73e51965d40370e209d7e9eed9c39f510d8348cbe95ab7b2"} err="failed to get container status \"654b952ec5cff6ad73e51965d40370e209d7e9eed9c39f510d8348cbe95ab7b2\": rpc error: code = NotFound desc = could not find container \"654b952ec5cff6ad73e51965d40370e209d7e9eed9c39f510d8348cbe95ab7b2\": container with ID starting with 654b952ec5cff6ad73e51965d40370e209d7e9eed9c39f510d8348cbe95ab7b2 not found: ID does not exist"
Nov 03 22:50:06.461670 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:06.461636    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv"]
Nov 03 22:50:06.473348 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:06.473319    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-j68rv"]
Nov 03 22:50:06.709232 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:06.709182    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:50:07.510043 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:07.510012    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:50:07.591345 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:07.591235    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-b023a2-f5bd44854-vjwb2"]
Nov 03 22:50:07.626031 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:07.591897    4431 cpu_manager.go:414] "containerMap: removing container" podUID="f8b1f74b-851b-41ed-a643-f01042642e59" containerName="istio-proxy"
Nov 03 22:50:07.626031 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:07.591908    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8b1f74b-851b-41ed-a643-f01042642e59" containerName="istio-proxy"
Nov 03 22:50:07.626031 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:07.592031    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8b1f74b-851b-41ed-a643-f01042642e59" containerName="istio-proxy"
Nov 03 22:50:07.626031 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:07.597372    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-b023a2-f5bd44854-vjwb2"
Nov 03 22:50:07.626031 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:07.623153    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w5-b023a2\""
Nov 03 22:50:07.634840 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:07.634810    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w5-b023a2-dockercfg-kxwg6\""
Nov 03 22:50:07.638537 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:07.638513    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-b023a2-f5bd44854-vjwb2"]
Nov 03 22:50:07.650677 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:07.650650    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9bfw\" (UniqueName: \"kubernetes.io/projected/444d998c-f1d3-46d1-8a77-89eded48361b-kube-api-access-g9bfw\") pod \"payload-processing-e2e-shared-a-w5-b023a2-f5bd44854-vjwb2\" (UID: \"444d998c-f1d3-46d1-8a77-89eded48361b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-b023a2-f5bd44854-vjwb2"
Nov 03 22:50:07.650937 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:07.650913    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/444d998c-f1d3-46d1-8a77-89eded48361b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-b023a2-f5bd44854-vjwb2\" (UID: \"444d998c-f1d3-46d1-8a77-89eded48361b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-b023a2-f5bd44854-vjwb2"
Nov 03 22:50:07.653545 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:07.653518    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:50:07.659792 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:07.659768    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-b023a2-678fcf5d79-x2g2r"]
Nov 03 22:50:07.665141 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:07.665121    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-b023a2-678fcf5d79-x2g2r"
Nov 03 22:50:07.673398 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:07.673373    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-b023a2-678fcf5d79-x2g2r"]
Nov 03 22:50:07.801105 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:07.801004    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9bfw\" (UniqueName: \"kubernetes.io/projected/444d998c-f1d3-46d1-8a77-89eded48361b-kube-api-access-g9bfw\") pod \"payload-processing-e2e-shared-a-w5-b023a2-f5bd44854-vjwb2\" (UID: \"444d998c-f1d3-46d1-8a77-89eded48361b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-b023a2-f5bd44854-vjwb2"
Nov 03 22:50:07.801338 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:07.801268    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4sv2\" (UniqueName: \"kubernetes.io/projected/2ae1c3c5-f16b-4e41-b70a-64999f5ab09b-kube-api-access-j4sv2\") pod \"payload-pre-processing-e2e-shared-a-w5-b023a2-678fcf5d79-x2g2r\" (UID: \"2ae1c3c5-f16b-4e41-b70a-64999f5ab09b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-b023a2-678fcf5d79-x2g2r"
Nov 03 22:50:07.801338 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:07.801322    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/444d998c-f1d3-46d1-8a77-89eded48361b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-b023a2-f5bd44854-vjwb2\" (UID: \"444d998c-f1d3-46d1-8a77-89eded48361b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-b023a2-f5bd44854-vjwb2"
Nov 03 22:50:07.801649 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:07.801506    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ae1c3c5-f16b-4e41-b70a-64999f5ab09b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-b023a2-678fcf5d79-x2g2r\" (UID: \"2ae1c3c5-f16b-4e41-b70a-64999f5ab09b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-b023a2-678fcf5d79-x2g2r"
Nov 03 22:50:07.802823 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:07.802760    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/444d998c-f1d3-46d1-8a77-89eded48361b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-b023a2-f5bd44854-vjwb2\" (UID: \"444d998c-f1d3-46d1-8a77-89eded48361b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-b023a2-f5bd44854-vjwb2"
Nov 03 22:50:07.829073 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:07.825240    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9bfw\" (UniqueName: \"kubernetes.io/projected/444d998c-f1d3-46d1-8a77-89eded48361b-kube-api-access-g9bfw\") pod \"payload-processing-e2e-shared-a-w5-b023a2-f5bd44854-vjwb2\" (UID: \"444d998c-f1d3-46d1-8a77-89eded48361b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-b023a2-f5bd44854-vjwb2"
Nov 03 22:50:07.902801 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:07.902763    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4sv2\" (UniqueName: \"kubernetes.io/projected/2ae1c3c5-f16b-4e41-b70a-64999f5ab09b-kube-api-access-j4sv2\") pod \"payload-pre-processing-e2e-shared-a-w5-b023a2-678fcf5d79-x2g2r\" (UID: \"2ae1c3c5-f16b-4e41-b70a-64999f5ab09b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-b023a2-678fcf5d79-x2g2r"
Nov 03 22:50:07.902976 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:07.902910    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ae1c3c5-f16b-4e41-b70a-64999f5ab09b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-b023a2-678fcf5d79-x2g2r\" (UID: \"2ae1c3c5-f16b-4e41-b70a-64999f5ab09b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-b023a2-678fcf5d79-x2g2r"
Nov 03 22:50:07.903678 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:07.903658    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ae1c3c5-f16b-4e41-b70a-64999f5ab09b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-b023a2-678fcf5d79-x2g2r\" (UID: \"2ae1c3c5-f16b-4e41-b70a-64999f5ab09b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-b023a2-678fcf5d79-x2g2r"
Nov 03 22:50:07.922567 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:07.922527    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-b023a2-f5bd44854-vjwb2"
Nov 03 22:50:07.925813 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:07.925782    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4sv2\" (UniqueName: \"kubernetes.io/projected/2ae1c3c5-f16b-4e41-b70a-64999f5ab09b-kube-api-access-j4sv2\") pod \"payload-pre-processing-e2e-shared-a-w5-b023a2-678fcf5d79-x2g2r\" (UID: \"2ae1c3c5-f16b-4e41-b70a-64999f5ab09b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-b023a2-678fcf5d79-x2g2r"
Nov 03 22:50:07.982846 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:07.982348    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-b023a2-678fcf5d79-x2g2r"
Nov 03 22:50:08.094387 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:08.092065    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8b1f74b-851b-41ed-a643-f01042642e59" path="/var/lib/kubelet/pods/f8b1f74b-851b-41ed-a643-f01042642e59/volumes"
Nov 03 22:50:08.233260 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:08.233229    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-b023a2-f5bd44854-vjwb2"]
Nov 03 22:50:08.234246 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:50:08.234187    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod444d998c_f1d3_46d1_8a77_89eded48361b.slice/crio-3d958bc2d8a3129c1b24ba9fef68605d1d535221ce721b6dfe08de9c443c0e77 WatchSource:0}: Error finding container 3d958bc2d8a3129c1b24ba9fef68605d1d535221ce721b6dfe08de9c443c0e77: Status 404 returned error can't find the container with id 3d958bc2d8a3129c1b24ba9fef68605d1d535221ce721b6dfe08de9c443c0e77
Nov 03 22:50:08.310626 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:08.310519    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-b023a2-678fcf5d79-x2g2r"]
Nov 03 22:50:08.310735 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:50:08.310701    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ae1c3c5_f16b_4e41_b70a_64999f5ab09b.slice/crio-7680755c303deb0aa014d258e234deaf881317f5054adcd2dc254c91e62ba7e7 WatchSource:0}: Error finding container 7680755c303deb0aa014d258e234deaf881317f5054adcd2dc254c91e62ba7e7: Status 404 returned error can't find the container with id 7680755c303deb0aa014d258e234deaf881317f5054adcd2dc254c91e62ba7e7
Nov 03 22:50:08.416789 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:08.416759    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:50:08.424982 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:08.424952    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-b023a2-678fcf5d79-x2g2r" event={"ID":"2ae1c3c5-f16b-4e41-b70a-64999f5ab09b","Type":"ContainerStarted","Data":"8740867a7cdc96212e7da8ea9e87c29dd2eea39b6f7119c7e11a579798da967b"}
Nov 03 22:50:08.425128 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:08.425003    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-b023a2-678fcf5d79-x2g2r" event={"ID":"2ae1c3c5-f16b-4e41-b70a-64999f5ab09b","Type":"ContainerStarted","Data":"7680755c303deb0aa014d258e234deaf881317f5054adcd2dc254c91e62ba7e7"}
Nov 03 22:50:08.425128 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:08.425026    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-b023a2-678fcf5d79-x2g2r"
Nov 03 22:50:08.426682 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:08.426661    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-b023a2-f5bd44854-vjwb2" event={"ID":"444d998c-f1d3-46d1-8a77-89eded48361b","Type":"ContainerStarted","Data":"40694285abbdc4f03139caa5c0632d85944095a3cd816759f7dbf875a5648ccf"}
Nov 03 22:50:08.426795 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:08.426705    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-b023a2-f5bd44854-vjwb2" event={"ID":"444d998c-f1d3-46d1-8a77-89eded48361b","Type":"ContainerStarted","Data":"3d958bc2d8a3129c1b24ba9fef68605d1d535221ce721b6dfe08de9c443c0e77"}
Nov 03 22:50:08.426795 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:08.426725    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-b023a2-f5bd44854-vjwb2"
Nov 03 22:50:08.476585 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:08.476521    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-b023a2-678fcf5d79-x2g2r" podStartSLOduration=1.476505299 podStartE2EDuration="1.476505299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:50:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:50:08.474442534 +0000 UTC m=+2232.500298395" watchObservedRunningTime="2026-11-03 22:50:08.476505299 +0000 UTC m=+2232.502361162"
Nov 03 22:50:08.515982 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:08.515920    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-b023a2-f5bd44854-vjwb2" podStartSLOduration=1.5159015519999999 podStartE2EDuration="1.515901552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:50:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:50:08.513874077 +0000 UTC m=+2232.539729938" watchObservedRunningTime="2026-11-03 22:50:08.515901552 +0000 UTC m=+2232.541757414"
Nov 03 22:50:10.064520 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:10.064487    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:50:11.556572 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:11.556533    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:50:13.022600 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:13.022566    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:50:14.564945 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:14.564905    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:50:16.122083 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:16.122045    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:50:17.619961 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:17.619918    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:50:19.114172 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:19.114116    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:50:19.437531 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:19.437504    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-b023a2-678fcf5d79-x2g2r"
Nov 03 22:50:19.437736 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:19.437577    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-b023a2-f5bd44854-vjwb2"
Nov 03 22:50:20.676130 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:20.676091    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:50:22.317299 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:22.317259    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:50:23.809640 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:23.809611    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:50:25.283254 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:25.283191    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:50:26.893431 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:26.893394    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:50:28.472603 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:28.472564    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:50:29.925734 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:29.925693    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:50:31.261166 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:50:31.261128    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:50:31.608454 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:31.608369    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:50:33.418950 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:33.418917    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:50:35.216129 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:35.216093    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:50:36.476508 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:36.476468    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:50:36.917067 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:36.917030    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:50:38.394905 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:38.394821    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:50:38.992389 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:38.992352    4431 patch_prober.go:29] interesting pod/e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 22:50:38.992564 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:38.992463    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t" podUID="fe16bcab-2fb0-425d-961c-adc59648bace" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:50:39.221725 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:39.221689    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:50:40.106029 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:40.105994    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:50:41.604905 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:41.604867    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:50:42.673530 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:42.673498    4431 generic.go:356] "Generic (PLEG): container finished" podID="fe16bcab-2fb0-425d-961c-adc59648bace" containerID="e0814691e68fe0fa1842c60813e59af9d1da0e745e0a14e3b15866d7a228c8c1" exitCode=0
Nov 03 22:50:42.673959 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:42.673569    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t" event={"ID":"fe16bcab-2fb0-425d-961c-adc59648bace","Type":"ContainerDied","Data":"e0814691e68fe0fa1842c60813e59af9d1da0e745e0a14e3b15866d7a228c8c1"}
Nov 03 22:50:42.674017 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:42.674002    4431 scope.go:122] "RemoveContainer" containerID="e0814691e68fe0fa1842c60813e59af9d1da0e745e0a14e3b15866d7a228c8c1"
Nov 03 22:50:42.675989 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:42.675963    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:50:42.676069 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:42.676039    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:50:42.676111 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:42.676072    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:50:43.193988 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:43.193955    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:50:43.682829 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:43.682788    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t" event={"ID":"fe16bcab-2fb0-425d-961c-adc59648bace","Type":"ContainerStarted","Data":"dfda9e3c3722843e060da24db1ea7b25c40cae65892de7873e46747a41991127"}
Nov 03 22:50:43.874629 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:43.874592    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t"
Nov 03 22:50:43.874826 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:43.874803    4431 patch_prober.go:29] interesting pod/e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Nov 03 22:50:43.874882 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:43.874851    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t" podUID="fe16bcab-2fb0-425d-961c-adc59648bace" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Nov 03 22:50:44.875152 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:44.875115    4431 patch_prober.go:29] interesting pod/e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Nov 03 22:50:44.875581 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:44.875181    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t" podUID="fe16bcab-2fb0-425d-961c-adc59648bace" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Nov 03 22:50:44.908466 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:44.908434    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:50:45.875572 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:45.875533    4431 patch_prober.go:29] interesting pod/e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Nov 03 22:50:45.876096 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:45.875598    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t" podUID="fe16bcab-2fb0-425d-961c-adc59648bace" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Nov 03 22:50:46.375391 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:46.375353    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:50:46.516098 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:46.516064    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:50:46.667873 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:46.667833    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:50:46.875131 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:46.875093    4431 patch_prober.go:29] interesting pod/e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Nov 03 22:50:46.875131 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:46.875155    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t" podUID="fe16bcab-2fb0-425d-961c-adc59648bace" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Nov 03 22:50:47.874924 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:47.874890    4431 patch_prober.go:29] interesting pod/e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Nov 03 22:50:47.875586 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:47.875538    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t" podUID="fe16bcab-2fb0-425d-961c-adc59648bace" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Nov 03 22:50:48.011639 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:48.011602    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:50:48.773701 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:48.773668    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:50:48.875012 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:48.874969    4431 patch_prober.go:29] interesting pod/e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Nov 03 22:50:48.875448 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:48.875036    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t" podUID="fe16bcab-2fb0-425d-961c-adc59648bace" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Nov 03 22:50:49.616446 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:49.616411    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:50:49.879310 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:49.879235    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t"
Nov 03 22:50:49.879725 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:49.879709    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t"
Nov 03 22:50:49.880415 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:49.880396    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t"
Nov 03 22:50:51.229664 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:51.229630    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:50:52.704207 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:52.704173    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:50:53.859302 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:53.859269    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:50:54.221422 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:54.221382    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:50:55.715679 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:55.715641    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:50:57.342733 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:57.342691    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:50:58.818896 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:50:58.818861    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:51:00.420068 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:00.420022    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:51:01.261116 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:51:01.261078    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:51:02.027617 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:02.027578    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:51:03.563010 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:03.562974    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:51:05.137292 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:05.137256    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:51:06.944389 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:06.944350    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:51:08.413684 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:08.413631    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:51:09.955623 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:09.955587    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:51:10.107190 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:10.104998    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:51:11.527554 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:11.527513    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:51:13.219125 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:13.219066    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:51:14.886413 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:14.886376    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:51:15.180899 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:15.180861    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-50b9b5/test-model-50b9b5-kserve-6c4d7fbdfc-clplg"]
Nov 03 22:51:15.212744 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:15.212708    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:51:16.444047 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:16.443999    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:51:17.923261 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:17.923204    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:51:19.731479 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:19.723614    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:51:21.253383 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:21.253352    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:51:22.782543 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:22.782506    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:51:24.429359 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:24.429319    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:51:26.005330 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:26.005294    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:51:27.308970 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:27.308934    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:51:27.600736 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:27.600655    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:51:27.658691 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:27.658652    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-fe7ca0/test-model-fe7ca0-kserve-5b69659fbc-jpdrg"]
Nov 03 22:51:27.788975 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:27.788939    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:51:29.295185 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:29.295150    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8"]
Nov 03 22:51:29.299048 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:29.299017    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8"
Nov 03 22:51:29.320511 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:29.320483    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8"]
Nov 03 22:51:29.327775 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:29.327748    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-d4b6bb-openshift-default-dockercfg-lvgrr\""
Nov 03 22:51:29.400929 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:29.400893    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c62c97b-1d80-4227-a469-08c2fa89b868-workload-certs\") pod \"e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8\" (UID: \"4c62c97b-1d80-4227-a469-08c2fa89b868\") " pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8"
Nov 03 22:51:29.400929 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:29.400940    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqpns\" (UniqueName: \"kubernetes.io/projected/4c62c97b-1d80-4227-a469-08c2fa89b868-kube-api-access-zqpns\") pod \"e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8\" (UID: \"4c62c97b-1d80-4227-a469-08c2fa89b868\") " pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8"
Nov 03 22:51:29.401147 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:29.401058    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c62c97b-1d80-4227-a469-08c2fa89b868-istio-token\") pod \"e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8\" (UID: \"4c62c97b-1d80-4227-a469-08c2fa89b868\") " pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8"
Nov 03 22:51:29.401147 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:29.401113    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c62c97b-1d80-4227-a469-08c2fa89b868-credential-socket\") pod \"e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8\" (UID: \"4c62c97b-1d80-4227-a469-08c2fa89b868\") " pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8"
Nov 03 22:51:29.401246 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:29.401177    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c62c97b-1d80-4227-a469-08c2fa89b868-istio-envoy\") pod \"e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8\" (UID: \"4c62c97b-1d80-4227-a469-08c2fa89b868\") " pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8"
Nov 03 22:51:29.401329 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:29.401310    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c62c97b-1d80-4227-a469-08c2fa89b868-istio-data\") pod \"e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8\" (UID: \"4c62c97b-1d80-4227-a469-08c2fa89b868\") " pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8"
Nov 03 22:51:29.401378 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:29.401365    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c62c97b-1d80-4227-a469-08c2fa89b868-workload-socket\") pod \"e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8\" (UID: \"4c62c97b-1d80-4227-a469-08c2fa89b868\") " pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8"
Nov 03 22:51:29.401421 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:29.401410    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c62c97b-1d80-4227-a469-08c2fa89b868-istiod-ca-cert\") pod \"e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8\" (UID: \"4c62c97b-1d80-4227-a469-08c2fa89b868\") " pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8"
Nov 03 22:51:29.401526 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:29.401512    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c62c97b-1d80-4227-a469-08c2fa89b868-istio-podinfo\") pod \"e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8\" (UID: \"4c62c97b-1d80-4227-a469-08c2fa89b868\") " pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8"
Nov 03 22:51:29.439029 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:29.438999    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:51:29.502473 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:29.502437    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c62c97b-1d80-4227-a469-08c2fa89b868-workload-socket\") pod \"e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8\" (UID: \"4c62c97b-1d80-4227-a469-08c2fa89b868\") " pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8"
Nov 03 22:51:29.502473 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:29.502476    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c62c97b-1d80-4227-a469-08c2fa89b868-istiod-ca-cert\") pod \"e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8\" (UID: \"4c62c97b-1d80-4227-a469-08c2fa89b868\") " pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8"
Nov 03 22:51:29.502699 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:29.502579    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c62c97b-1d80-4227-a469-08c2fa89b868-istio-podinfo\") pod \"e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8\" (UID: \"4c62c97b-1d80-4227-a469-08c2fa89b868\") " pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8"
Nov 03 22:51:29.502699 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:29.502610    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c62c97b-1d80-4227-a469-08c2fa89b868-workload-certs\") pod \"e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8\" (UID: \"4c62c97b-1d80-4227-a469-08c2fa89b868\") " pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8"
Nov 03 22:51:29.502699 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:29.502632    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqpns\" (UniqueName: \"kubernetes.io/projected/4c62c97b-1d80-4227-a469-08c2fa89b868-kube-api-access-zqpns\") pod \"e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8\" (UID: \"4c62c97b-1d80-4227-a469-08c2fa89b868\") " pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8"
Nov 03 22:51:29.502859 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:29.502710    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c62c97b-1d80-4227-a469-08c2fa89b868-istio-token\") pod \"e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8\" (UID: \"4c62c97b-1d80-4227-a469-08c2fa89b868\") " pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8"
Nov 03 22:51:29.502859 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:29.502772    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c62c97b-1d80-4227-a469-08c2fa89b868-credential-socket\") pod \"e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8\" (UID: \"4c62c97b-1d80-4227-a469-08c2fa89b868\") " pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8"
Nov 03 22:51:29.502859 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:29.502836    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c62c97b-1d80-4227-a469-08c2fa89b868-istio-envoy\") pod \"e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8\" (UID: \"4c62c97b-1d80-4227-a469-08c2fa89b868\") " pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8"
Nov 03 22:51:29.503000 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:29.502905    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c62c97b-1d80-4227-a469-08c2fa89b868-istio-data\") pod \"e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8\" (UID: \"4c62c97b-1d80-4227-a469-08c2fa89b868\") " pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8"
Nov 03 22:51:29.503000 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:29.502922    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c62c97b-1d80-4227-a469-08c2fa89b868-workload-socket\") pod \"e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8\" (UID: \"4c62c97b-1d80-4227-a469-08c2fa89b868\") " pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8"
Nov 03 22:51:29.503133 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:29.503002    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c62c97b-1d80-4227-a469-08c2fa89b868-workload-certs\") pod \"e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8\" (UID: \"4c62c97b-1d80-4227-a469-08c2fa89b868\") " pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8"
Nov 03 22:51:29.503323 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:29.503303    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c62c97b-1d80-4227-a469-08c2fa89b868-istio-data\") pod \"e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8\" (UID: \"4c62c97b-1d80-4227-a469-08c2fa89b868\") " pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8"
Nov 03 22:51:29.503407 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:29.503319    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c62c97b-1d80-4227-a469-08c2fa89b868-istiod-ca-cert\") pod \"e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8\" (UID: \"4c62c97b-1d80-4227-a469-08c2fa89b868\") " pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8"
Nov 03 22:51:29.503407 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:29.503331    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c62c97b-1d80-4227-a469-08c2fa89b868-credential-socket\") pod \"e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8\" (UID: \"4c62c97b-1d80-4227-a469-08c2fa89b868\") " pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8"
Nov 03 22:51:29.505172 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:29.505154    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c62c97b-1d80-4227-a469-08c2fa89b868-istio-podinfo\") pod \"e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8\" (UID: \"4c62c97b-1d80-4227-a469-08c2fa89b868\") " pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8"
Nov 03 22:51:29.505379 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:29.505363    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c62c97b-1d80-4227-a469-08c2fa89b868-istio-envoy\") pod \"e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8\" (UID: \"4c62c97b-1d80-4227-a469-08c2fa89b868\") " pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8"
Nov 03 22:51:29.525883 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:29.525856    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c62c97b-1d80-4227-a469-08c2fa89b868-istio-token\") pod \"e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8\" (UID: \"4c62c97b-1d80-4227-a469-08c2fa89b868\") " pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8"
Nov 03 22:51:29.525998 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:29.525913    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqpns\" (UniqueName: \"kubernetes.io/projected/4c62c97b-1d80-4227-a469-08c2fa89b868-kube-api-access-zqpns\") pod \"e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8\" (UID: \"4c62c97b-1d80-4227-a469-08c2fa89b868\") " pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8"
Nov 03 22:51:29.613270 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:29.613165    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8"
Nov 03 22:51:29.935301 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:29.935274    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8"]
Nov 03 22:51:29.935934 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:51:29.935914    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c62c97b_1d80_4227_a469_08c2fa89b868.slice/crio-3e502a6b934f6211938d72064f99b5c9b79b4eada7e879f898a8a41cd3ebf873 WatchSource:0}: Error finding container 3e502a6b934f6211938d72064f99b5c9b79b4eada7e879f898a8a41cd3ebf873: Status 404 returned error can't find the container with id 3e502a6b934f6211938d72064f99b5c9b79b4eada7e879f898a8a41cd3ebf873
Nov 03 22:51:29.938846 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:29.938812    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:51:29.938934 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:29.938883    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:51:29.938934 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:29.938909    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:51:30.066860 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:30.066825    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8" event={"ID":"4c62c97b-1d80-4227-a469-08c2fa89b868","Type":"ContainerStarted","Data":"6d971e3ff060b1f0171d344da36a60b908f6dcd39c4857e163dbad14cba84649"}
Nov 03 22:51:30.067045 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:30.066883    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8" event={"ID":"4c62c97b-1d80-4227-a469-08c2fa89b868","Type":"ContainerStarted","Data":"3e502a6b934f6211938d72064f99b5c9b79b4eada7e879f898a8a41cd3ebf873"}
Nov 03 22:51:30.118540 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:30.118476    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8" podStartSLOduration=1.118455228 podStartE2EDuration="1.118455228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:51:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:51:30.114852737 +0000 UTC m=+2314.140708600" watchObservedRunningTime="2026-11-03 22:51:30.118455228 +0000 UTC m=+2314.144311092"
Nov 03 22:51:30.208336 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:30.208232    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-50b9b5/test-model-50b9b5-kserve-6c4d7fbdfc-clplg" podUID="79bf63c8-226a-410a-933e-c97971c11629" containerName="main" containerID="cri-o://45d4762d98d546502e6b26196aa08fec11a8fa4f658bc83be456a7f897f67aad" gracePeriod=45
Nov 03 22:51:30.347537 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:30.347239    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-50b9b5/test-model-50b9b5-kserve-6c4d7fbdfc-clplg" podUID="79bf63c8-226a-410a-933e-c97971c11629" containerName="main" probeResult="failure" output="Get \"https://10.133.0.123:8000/ready\": dial tcp 10.133.0.123:8000: connect: connection refused"
Nov 03 22:51:30.472574 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:30.472545    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-50b9b5/test-model-50b9b5-kserve-6c4d7fbdfc-clplg"
Nov 03 22:51:30.533133 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:30.532969    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79bf63c8-226a-410a-933e-c97971c11629-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79bf63c8-226a-410a-933e-c97971c11629-model-cache\") pod \"79bf63c8-226a-410a-933e-c97971c11629\" (UID: \"79bf63c8-226a-410a-933e-c97971c11629\") "
Nov 03 22:51:30.533841 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:30.533242    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79bf63c8-226a-410a-933e-c97971c11629-tls-certs\" (UniqueName: \"kubernetes.io/secret/79bf63c8-226a-410a-933e-c97971c11629-tls-certs\") pod \"79bf63c8-226a-410a-933e-c97971c11629\" (UID: \"79bf63c8-226a-410a-933e-c97971c11629\") "
Nov 03 22:51:30.533841 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:30.533333    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79bf63c8-226a-410a-933e-c97971c11629-home\" (UniqueName: \"kubernetes.io/empty-dir/79bf63c8-226a-410a-933e-c97971c11629-home\") pod \"79bf63c8-226a-410a-933e-c97971c11629\" (UID: \"79bf63c8-226a-410a-933e-c97971c11629\") "
Nov 03 22:51:30.533841 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:30.533408    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79bf63c8-226a-410a-933e-c97971c11629-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79bf63c8-226a-410a-933e-c97971c11629-tmp-dir\") pod \"79bf63c8-226a-410a-933e-c97971c11629\" (UID: \"79bf63c8-226a-410a-933e-c97971c11629\") "
Nov 03 22:51:30.533841 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:30.533447    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79bf63c8-226a-410a-933e-c97971c11629-kube-api-access-bq4c8\" (UniqueName: \"kubernetes.io/projected/79bf63c8-226a-410a-933e-c97971c11629-kube-api-access-bq4c8\") pod \"79bf63c8-226a-410a-933e-c97971c11629\" (UID: \"79bf63c8-226a-410a-933e-c97971c11629\") "
Nov 03 22:51:30.533841 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:30.533516    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79bf63c8-226a-410a-933e-c97971c11629-dshm\" (UniqueName: \"kubernetes.io/empty-dir/79bf63c8-226a-410a-933e-c97971c11629-dshm\") pod \"79bf63c8-226a-410a-933e-c97971c11629\" (UID: \"79bf63c8-226a-410a-933e-c97971c11629\") "
Nov 03 22:51:30.534562 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:30.534528    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79bf63c8-226a-410a-933e-c97971c11629-home" pod "79bf63c8-226a-410a-933e-c97971c11629" (UID: "79bf63c8-226a-410a-933e-c97971c11629"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:51:30.534704 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:30.534581    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79bf63c8-226a-410a-933e-c97971c11629-model-cache" pod "79bf63c8-226a-410a-933e-c97971c11629" (UID: "79bf63c8-226a-410a-933e-c97971c11629"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:51:30.534704 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:30.534644    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79bf63c8-226a-410a-933e-c97971c11629-tmp-dir" pod "79bf63c8-226a-410a-933e-c97971c11629" (UID: "79bf63c8-226a-410a-933e-c97971c11629"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:51:30.539029 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:30.538982    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79bf63c8-226a-410a-933e-c97971c11629-kube-api-access-bq4c8" pod "79bf63c8-226a-410a-933e-c97971c11629" (UID: "79bf63c8-226a-410a-933e-c97971c11629"). InnerVolumeSpecName "kube-api-access-bq4c8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:51:30.579575 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:30.579533    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79bf63c8-226a-410a-933e-c97971c11629-dshm" pod "79bf63c8-226a-410a-933e-c97971c11629" (UID: "79bf63c8-226a-410a-933e-c97971c11629"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:51:30.579768 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:30.579648    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79bf63c8-226a-410a-933e-c97971c11629-tls-certs" pod "79bf63c8-226a-410a-933e-c97971c11629" (UID: "79bf63c8-226a-410a-933e-c97971c11629"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:51:30.614161 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:30.614130    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8"
Nov 03 22:51:30.635798 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:30.635770    4431 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79bf63c8-226a-410a-933e-c97971c11629-model-cache\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:51:30.635798 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:30.635797    4431 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79bf63c8-226a-410a-933e-c97971c11629-tls-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:51:30.635798 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:30.635806    4431 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79bf63c8-226a-410a-933e-c97971c11629-home\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:51:30.636007 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:30.635814    4431 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79bf63c8-226a-410a-933e-c97971c11629-tmp-dir\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:51:30.636007 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:30.635824    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bq4c8\" (UniqueName: \"kubernetes.io/projected/79bf63c8-226a-410a-933e-c97971c11629-kube-api-access-bq4c8\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:51:30.636007 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:30.635832    4431 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79bf63c8-226a-410a-933e-c97971c11629-dshm\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:51:31.067971 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:31.067936    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:51:31.080612 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:31.079257    4431 generic.go:356] "Generic (PLEG): container finished" podID="79bf63c8-226a-410a-933e-c97971c11629" containerID="45d4762d98d546502e6b26196aa08fec11a8fa4f658bc83be456a7f897f67aad" exitCode=0
Nov 03 22:51:31.093818 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:31.081323    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-50b9b5/test-model-50b9b5-kserve-6c4d7fbdfc-clplg"
Nov 03 22:51:31.093818 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:31.089769    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-50b9b5/test-model-50b9b5-kserve-6c4d7fbdfc-clplg" event={"ID":"79bf63c8-226a-410a-933e-c97971c11629","Type":"ContainerDied","Data":"45d4762d98d546502e6b26196aa08fec11a8fa4f658bc83be456a7f897f67aad"}
Nov 03 22:51:31.093818 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:31.089846    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-50b9b5/test-model-50b9b5-kserve-6c4d7fbdfc-clplg" event={"ID":"79bf63c8-226a-410a-933e-c97971c11629","Type":"ContainerDied","Data":"298d7f4c549ba000c1953eb99183aecb34b9a2f570f7411185a6124854602316"}
Nov 03 22:51:31.093818 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:31.089872    4431 scope.go:122] "RemoveContainer" containerID="45d4762d98d546502e6b26196aa08fec11a8fa4f658bc83be456a7f897f67aad"
Nov 03 22:51:31.126099 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:31.126074    4431 scope.go:122] "RemoveContainer" containerID="45d4762d98d546502e6b26196aa08fec11a8fa4f658bc83be456a7f897f67aad"
Nov 03 22:51:31.126786 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:51:31.126761    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45d4762d98d546502e6b26196aa08fec11a8fa4f658bc83be456a7f897f67aad\": container with ID starting with 45d4762d98d546502e6b26196aa08fec11a8fa4f658bc83be456a7f897f67aad not found: ID does not exist" containerID="45d4762d98d546502e6b26196aa08fec11a8fa4f658bc83be456a7f897f67aad"
Nov 03 22:51:31.126900 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:31.126792    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45d4762d98d546502e6b26196aa08fec11a8fa4f658bc83be456a7f897f67aad"} err="failed to get container status \"45d4762d98d546502e6b26196aa08fec11a8fa4f658bc83be456a7f897f67aad\": rpc error: code = NotFound desc = could not find container \"45d4762d98d546502e6b26196aa08fec11a8fa4f658bc83be456a7f897f67aad\": container with ID starting with 45d4762d98d546502e6b26196aa08fec11a8fa4f658bc83be456a7f897f67aad not found: ID does not exist"
Nov 03 22:51:31.189803 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:31.189767    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-50b9b5/test-model-50b9b5-kserve-6c4d7fbdfc-clplg"]
Nov 03 22:51:31.216574 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:31.216539    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-50b9b5/test-model-50b9b5-kserve-6c4d7fbdfc-clplg"]
Nov 03 22:51:31.284573 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:51:31.284537    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:51:31.615815 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:31.615780    4431 patch_prober.go:29] interesting pod/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 03 22:51:31.616301 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:31.615832    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8" podUID="4c62c97b-1d80-4227-a469-08c2fa89b868" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 03 22:51:32.090607 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:32.090571    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79bf63c8-226a-410a-933e-c97971c11629" path="/var/lib/kubelet/pods/79bf63c8-226a-410a-933e-c97971c11629/volumes"
Nov 03 22:51:32.614389 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:32.614357    4431 patch_prober.go:29] interesting pod/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 03 22:51:32.614567 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:32.614417    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8" podUID="4c62c97b-1d80-4227-a469-08c2fa89b868" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 03 22:51:32.801422 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:32.801385    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:51:32.954253 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:32.948717    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:51:33.252293 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:33.250895    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:51:33.613914 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:33.613840    4431 patch_prober.go:29] interesting pod/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 03 22:51:33.614122 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:33.614100    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8" podUID="4c62c97b-1d80-4227-a469-08c2fa89b868" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 03 22:51:33.982990 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:33.982956    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:51:34.142825 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:34.142790    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:51:34.484872 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:34.484830    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:51:34.614177 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:34.614148    4431 patch_prober.go:29] interesting pod/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 03 22:51:34.614371 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:34.614196    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8" podUID="4c62c97b-1d80-4227-a469-08c2fa89b868" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 03 22:51:35.613932 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:35.613901    4431 patch_prober.go:29] interesting pod/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 03 22:51:35.614367 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:35.613949    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8" podUID="4c62c97b-1d80-4227-a469-08c2fa89b868" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 03 22:51:36.096229 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:36.096171    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:51:36.619036 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:36.619005    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8"
Nov 03 22:51:36.619554 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:36.619410    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8"
Nov 03 22:51:36.620290 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:36.620268    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8"
Nov 03 22:51:37.708081 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:37.708046    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:51:37.902252 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:37.897932    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:51:38.186089 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:38.186055    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:51:38.351671 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:38.351637    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:51:38.787018 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:38.786985    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-50b9b5-7f48f98986-ddh6w"]
Nov 03 22:51:38.990033 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:38.990003    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:51:39.139355 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:39.139274    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:51:39.708760 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:39.708723    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:51:40.158998 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:40.158913    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:51:40.232825 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:40.232792    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-9b841d-5c95644856-295pk"]
Nov 03 22:51:40.233554 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:40.233536    4431 cpu_manager.go:414] "containerMap: removing container" podUID="79bf63c8-226a-410a-933e-c97971c11629" containerName="main"
Nov 03 22:51:40.233616 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:40.233555    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79bf63c8-226a-410a-933e-c97971c11629" containerName="main"
Nov 03 22:51:40.233680 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:40.233670    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="79bf63c8-226a-410a-933e-c97971c11629" containerName="main"
Nov 03 22:51:40.237252 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:40.237229    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-9b841d-5c95644856-295pk"
Nov 03 22:51:40.257910 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:40.257876    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-9b841d-5c95644856-295pk"]
Nov 03 22:51:40.262147 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:40.262122    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w5-9b841d\""
Nov 03 22:51:40.298055 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:40.298021    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w5-9b841d-dockercfg-z66wz\""
Nov 03 22:51:40.303958 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:40.303926    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:51:40.347631 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:40.347600    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n65qs\" (UniqueName: \"kubernetes.io/projected/aa0f4c59-fbe8-4b5c-b2fb-452653ac95b9-kube-api-access-n65qs\") pod \"payload-processing-e2e-shared-b-w5-9b841d-5c95644856-295pk\" (UID: \"aa0f4c59-fbe8-4b5c-b2fb-452653ac95b9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-9b841d-5c95644856-295pk"
Nov 03 22:51:40.347793 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:40.347720    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa0f4c59-fbe8-4b5c-b2fb-452653ac95b9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-9b841d-5c95644856-295pk\" (UID: \"aa0f4c59-fbe8-4b5c-b2fb-452653ac95b9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-9b841d-5c95644856-295pk"
Nov 03 22:51:40.456239 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:40.450990    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa0f4c59-fbe8-4b5c-b2fb-452653ac95b9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-9b841d-5c95644856-295pk\" (UID: \"aa0f4c59-fbe8-4b5c-b2fb-452653ac95b9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-9b841d-5c95644856-295pk"
Nov 03 22:51:40.456239 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:40.451317    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n65qs\" (UniqueName: \"kubernetes.io/projected/aa0f4c59-fbe8-4b5c-b2fb-452653ac95b9-kube-api-access-n65qs\") pod \"payload-processing-e2e-shared-b-w5-9b841d-5c95644856-295pk\" (UID: \"aa0f4c59-fbe8-4b5c-b2fb-452653ac95b9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-9b841d-5c95644856-295pk"
Nov 03 22:51:40.456239 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:40.452792    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa0f4c59-fbe8-4b5c-b2fb-452653ac95b9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-9b841d-5c95644856-295pk\" (UID: \"aa0f4c59-fbe8-4b5c-b2fb-452653ac95b9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-9b841d-5c95644856-295pk"
Nov 03 22:51:40.491953 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:40.491889    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n65qs\" (UniqueName: \"kubernetes.io/projected/aa0f4c59-fbe8-4b5c-b2fb-452653ac95b9-kube-api-access-n65qs\") pod \"payload-processing-e2e-shared-b-w5-9b841d-5c95644856-295pk\" (UID: \"aa0f4c59-fbe8-4b5c-b2fb-452653ac95b9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-9b841d-5c95644856-295pk"
Nov 03 22:51:40.559729 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:40.559699    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-9b841d-5c95644856-295pk"
Nov 03 22:51:40.861799 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:40.861771    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-9b841d-5c95644856-295pk"]
Nov 03 22:51:40.863164 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:51:40.863136    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa0f4c59_fbe8_4b5c_b2fb_452653ac95b9.slice/crio-713602736abb3dd5d720ef9f7ebd411fb543758239f1bdaf64427001b848b745 WatchSource:0}: Error finding container 713602736abb3dd5d720ef9f7ebd411fb543758239f1bdaf64427001b848b745: Status 404 returned error can't find the container with id 713602736abb3dd5d720ef9f7ebd411fb543758239f1bdaf64427001b848b745
Nov 03 22:51:41.183244 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:41.183179    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-9b841d-5c95644856-295pk" event={"ID":"aa0f4c59-fbe8-4b5c-b2fb-452653ac95b9","Type":"ContainerStarted","Data":"c8b1a2afbf9c7c8a81702d42f2696c5193b771830283f9b6cf505bffbb991a5f"}
Nov 03 22:51:41.183633 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:41.183262    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-9b841d-5c95644856-295pk" event={"ID":"aa0f4c59-fbe8-4b5c-b2fb-452653ac95b9","Type":"ContainerStarted","Data":"713602736abb3dd5d720ef9f7ebd411fb543758239f1bdaf64427001b848b745"}
Nov 03 22:51:41.183633 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:41.183293    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-9b841d-5c95644856-295pk"
Nov 03 22:51:41.223064 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:41.223009    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-9b841d-5c95644856-295pk" podStartSLOduration=1.2229912139999999 podStartE2EDuration="1.222991214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:51:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:51:41.222300429 +0000 UTC m=+2325.248156291" watchObservedRunningTime="2026-11-03 22:51:41.222991214 +0000 UTC m=+2325.248847075"
Nov 03 22:51:41.526590 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:41.526507    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:51:42.681131 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:42.681080    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-fe7ca0/test-model-fe7ca0-kserve-5b69659fbc-jpdrg" podUID="4f04425d-fb46-454f-84af-dd29b0208c21" containerName="main" containerID="cri-o://532e3fd87d90edf7a10ac1656eabbc2280231647a15a4a9a5743cadd305a8dc1" gracePeriod=45
Nov 03 22:51:42.957882 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:42.957853    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-fe7ca0/test-model-fe7ca0-kserve-5b69659fbc-jpdrg"
Nov 03 22:51:42.985524 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:42.985492    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f04425d-fb46-454f-84af-dd29b0208c21-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f04425d-fb46-454f-84af-dd29b0208c21-model-cache\") pod \"4f04425d-fb46-454f-84af-dd29b0208c21\" (UID: \"4f04425d-fb46-454f-84af-dd29b0208c21\") "
Nov 03 22:51:42.985689 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:42.985605    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f04425d-fb46-454f-84af-dd29b0208c21-tls-certs\" (UniqueName: \"kubernetes.io/secret/4f04425d-fb46-454f-84af-dd29b0208c21-tls-certs\") pod \"4f04425d-fb46-454f-84af-dd29b0208c21\" (UID: \"4f04425d-fb46-454f-84af-dd29b0208c21\") "
Nov 03 22:51:42.985689 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:42.985642    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f04425d-fb46-454f-84af-dd29b0208c21-kube-api-access-sn6r7\" (UniqueName: \"kubernetes.io/projected/4f04425d-fb46-454f-84af-dd29b0208c21-kube-api-access-sn6r7\") pod \"4f04425d-fb46-454f-84af-dd29b0208c21\" (UID: \"4f04425d-fb46-454f-84af-dd29b0208c21\") "
Nov 03 22:51:42.985811 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:42.985771    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f04425d-fb46-454f-84af-dd29b0208c21-model-cache" pod "4f04425d-fb46-454f-84af-dd29b0208c21" (UID: "4f04425d-fb46-454f-84af-dd29b0208c21"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:51:42.985871 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:42.985857    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f04425d-fb46-454f-84af-dd29b0208c21-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f04425d-fb46-454f-84af-dd29b0208c21-tmp-dir\") pod \"4f04425d-fb46-454f-84af-dd29b0208c21\" (UID: \"4f04425d-fb46-454f-84af-dd29b0208c21\") "
Nov 03 22:51:42.985930 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:42.985899    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f04425d-fb46-454f-84af-dd29b0208c21-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f04425d-fb46-454f-84af-dd29b0208c21-dshm\") pod \"4f04425d-fb46-454f-84af-dd29b0208c21\" (UID: \"4f04425d-fb46-454f-84af-dd29b0208c21\") "
Nov 03 22:51:42.985930 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:42.985927    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f04425d-fb46-454f-84af-dd29b0208c21-home\" (UniqueName: \"kubernetes.io/empty-dir/4f04425d-fb46-454f-84af-dd29b0208c21-home\") pod \"4f04425d-fb46-454f-84af-dd29b0208c21\" (UID: \"4f04425d-fb46-454f-84af-dd29b0208c21\") "
Nov 03 22:51:42.986134 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:42.986104    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f04425d-fb46-454f-84af-dd29b0208c21-tmp-dir" pod "4f04425d-fb46-454f-84af-dd29b0208c21" (UID: "4f04425d-fb46-454f-84af-dd29b0208c21"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:51:42.986279 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:42.986258    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f04425d-fb46-454f-84af-dd29b0208c21-home" pod "4f04425d-fb46-454f-84af-dd29b0208c21" (UID: "4f04425d-fb46-454f-84af-dd29b0208c21"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:51:42.986565 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:42.986546    4431 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f04425d-fb46-454f-84af-dd29b0208c21-tmp-dir\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:51:42.986565 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:42.986565    4431 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f04425d-fb46-454f-84af-dd29b0208c21-home\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:51:42.986710 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:42.986574    4431 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f04425d-fb46-454f-84af-dd29b0208c21-model-cache\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:51:42.987994 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:42.987970    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f04425d-fb46-454f-84af-dd29b0208c21-kube-api-access-sn6r7" pod "4f04425d-fb46-454f-84af-dd29b0208c21" (UID: "4f04425d-fb46-454f-84af-dd29b0208c21"). InnerVolumeSpecName "kube-api-access-sn6r7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:51:42.988087 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:42.987998    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f04425d-fb46-454f-84af-dd29b0208c21-tls-certs" pod "4f04425d-fb46-454f-84af-dd29b0208c21" (UID: "4f04425d-fb46-454f-84af-dd29b0208c21"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:51:42.988087 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:42.988021    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f04425d-fb46-454f-84af-dd29b0208c21-dshm" pod "4f04425d-fb46-454f-84af-dd29b0208c21" (UID: "4f04425d-fb46-454f-84af-dd29b0208c21"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:51:43.087811 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:43.087771    4431 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f04425d-fb46-454f-84af-dd29b0208c21-dshm\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:51:43.087969 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:43.087900    4431 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f04425d-fb46-454f-84af-dd29b0208c21-tls-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:51:43.087969 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:43.087928    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sn6r7\" (UniqueName: \"kubernetes.io/projected/4f04425d-fb46-454f-84af-dd29b0208c21-kube-api-access-sn6r7\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:51:43.198706 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:43.198627    4431 generic.go:356] "Generic (PLEG): container finished" podID="4f04425d-fb46-454f-84af-dd29b0208c21" containerID="532e3fd87d90edf7a10ac1656eabbc2280231647a15a4a9a5743cadd305a8dc1" exitCode=0
Nov 03 22:51:43.198706 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:43.198670    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-fe7ca0/test-model-fe7ca0-kserve-5b69659fbc-jpdrg"
Nov 03 22:51:43.198888 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:43.198707    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-fe7ca0/test-model-fe7ca0-kserve-5b69659fbc-jpdrg" event={"ID":"4f04425d-fb46-454f-84af-dd29b0208c21","Type":"ContainerDied","Data":"532e3fd87d90edf7a10ac1656eabbc2280231647a15a4a9a5743cadd305a8dc1"}
Nov 03 22:51:43.198888 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:43.198753    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-fe7ca0/test-model-fe7ca0-kserve-5b69659fbc-jpdrg" event={"ID":"4f04425d-fb46-454f-84af-dd29b0208c21","Type":"ContainerDied","Data":"b69cdc8840803a05865a71af65b673d727909a1baf08837dad96e7dd418c9ade"}
Nov 03 22:51:43.198888 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:43.198773    4431 scope.go:122] "RemoveContainer" containerID="532e3fd87d90edf7a10ac1656eabbc2280231647a15a4a9a5743cadd305a8dc1"
Nov 03 22:51:43.210725 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:43.210630    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:51:43.211048 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:43.211028    4431 scope.go:122] "RemoveContainer" containerID="532e3fd87d90edf7a10ac1656eabbc2280231647a15a4a9a5743cadd305a8dc1"
Nov 03 22:51:43.211437 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:51:43.211400    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"532e3fd87d90edf7a10ac1656eabbc2280231647a15a4a9a5743cadd305a8dc1\": container with ID starting with 532e3fd87d90edf7a10ac1656eabbc2280231647a15a4a9a5743cadd305a8dc1 not found: ID does not exist" containerID="532e3fd87d90edf7a10ac1656eabbc2280231647a15a4a9a5743cadd305a8dc1"
Nov 03 22:51:43.211533 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:43.211448    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"532e3fd87d90edf7a10ac1656eabbc2280231647a15a4a9a5743cadd305a8dc1"} err="failed to get container status \"532e3fd87d90edf7a10ac1656eabbc2280231647a15a4a9a5743cadd305a8dc1\": rpc error: code = NotFound desc = could not find container \"532e3fd87d90edf7a10ac1656eabbc2280231647a15a4a9a5743cadd305a8dc1\": container with ID starting with 532e3fd87d90edf7a10ac1656eabbc2280231647a15a4a9a5743cadd305a8dc1 not found: ID does not exist"
Nov 03 22:51:43.254842 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:43.254810    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-fe7ca0/test-model-fe7ca0-kserve-5b69659fbc-jpdrg"]
Nov 03 22:51:43.259754 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:43.259724    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-fe7ca0/test-model-fe7ca0-kserve-5b69659fbc-jpdrg"]
Nov 03 22:51:43.787885 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:43.787830    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-50b9b5-7f48f98986-ddh6w" podUID="5f000cab-17f2-4a9f-a463-54983bba8566" containerName="payload-processing" containerID="cri-o://fd1b2dc4dced9aba9d1592f9840982e16c9e7e4f336caf7d7cd01e2e23ad6d07" gracePeriod=25
Nov 03 22:51:44.089631 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:44.089544    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f04425d-fb46-454f-84af-dd29b0208c21" path="/var/lib/kubelet/pods/4f04425d-fb46-454f-84af-dd29b0208c21/volumes"
Nov 03 22:51:44.595181 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:44.595144    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-50b9b5-7f48f98986-ddh6w" podUID="5f000cab-17f2-4a9f-a463-54983bba8566" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 22:51:44.709376 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:44.707977    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:51:46.170678 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:46.170644    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:51:46.610154 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:46.610060    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:51:47.777084 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:47.777051    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:51:48.906140 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:48.906105    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:51:48.974305 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:48.974269    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7940ea-565f5768b6-8nzcp"]
Nov 03 22:51:48.974580 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:48.974551    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7940ea-565f5768b6-8nzcp" podUID="789373aa-d138-4dff-a505-17410f8cf0c7" containerName="maas-api" containerID="cri-o://5e85065346e882020a2dca2a640cbad3a08b96ee865fcc60d3c6febf1c129745" gracePeriod=30
Nov 03 22:51:49.068276 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:49.068183    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:51:49.312299 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:49.311905    4431 generic.go:356] "Generic (PLEG): container finished" podID="789373aa-d138-4dff-a505-17410f8cf0c7" containerID="5e85065346e882020a2dca2a640cbad3a08b96ee865fcc60d3c6febf1c129745" exitCode=0
Nov 03 22:51:49.312299 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:49.312100    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7940ea-565f5768b6-8nzcp" event={"ID":"789373aa-d138-4dff-a505-17410f8cf0c7","Type":"ContainerDied","Data":"5e85065346e882020a2dca2a640cbad3a08b96ee865fcc60d3c6febf1c129745"}
Nov 03 22:51:49.455241 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:49.444842    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7940ea-565f5768b6-8nzcp"
Nov 03 22:51:49.514984 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:49.514950    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/789373aa-d138-4dff-a505-17410f8cf0c7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/789373aa-d138-4dff-a505-17410f8cf0c7-openshift-service-ca\") pod \"789373aa-d138-4dff-a505-17410f8cf0c7\" (UID: \"789373aa-d138-4dff-a505-17410f8cf0c7\") "
Nov 03 22:51:49.515193 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:49.515021    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/789373aa-d138-4dff-a505-17410f8cf0c7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/789373aa-d138-4dff-a505-17410f8cf0c7-metrics-certs\") pod \"789373aa-d138-4dff-a505-17410f8cf0c7\" (UID: \"789373aa-d138-4dff-a505-17410f8cf0c7\") "
Nov 03 22:51:49.515193 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:49.515173    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/789373aa-d138-4dff-a505-17410f8cf0c7-kube-api-access-sb6vk\" (UniqueName: \"kubernetes.io/projected/789373aa-d138-4dff-a505-17410f8cf0c7-kube-api-access-sb6vk\") pod \"789373aa-d138-4dff-a505-17410f8cf0c7\" (UID: \"789373aa-d138-4dff-a505-17410f8cf0c7\") "
Nov 03 22:51:49.515394 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:49.515371    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/789373aa-d138-4dff-a505-17410f8cf0c7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/789373aa-d138-4dff-a505-17410f8cf0c7-maas-api-tls\") pod \"789373aa-d138-4dff-a505-17410f8cf0c7\" (UID: \"789373aa-d138-4dff-a505-17410f8cf0c7\") "
Nov 03 22:51:49.515462 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:49.515393    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/789373aa-d138-4dff-a505-17410f8cf0c7-openshift-service-ca" pod "789373aa-d138-4dff-a505-17410f8cf0c7" (UID: "789373aa-d138-4dff-a505-17410f8cf0c7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:51:49.516055 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:49.516020    4431 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/789373aa-d138-4dff-a505-17410f8cf0c7-openshift-service-ca\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:51:49.518660 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:49.518635    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/789373aa-d138-4dff-a505-17410f8cf0c7-maas-api-tls" pod "789373aa-d138-4dff-a505-17410f8cf0c7" (UID: "789373aa-d138-4dff-a505-17410f8cf0c7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:51:49.519549 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:49.519528    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/789373aa-d138-4dff-a505-17410f8cf0c7-metrics-certs" pod "789373aa-d138-4dff-a505-17410f8cf0c7" (UID: "789373aa-d138-4dff-a505-17410f8cf0c7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:51:49.521341 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:49.521321    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/789373aa-d138-4dff-a505-17410f8cf0c7-kube-api-access-sb6vk" pod "789373aa-d138-4dff-a505-17410f8cf0c7" (UID: "789373aa-d138-4dff-a505-17410f8cf0c7"). InnerVolumeSpecName "kube-api-access-sb6vk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:51:49.587547 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:49.587512    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:51:49.618894 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:49.617580    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sb6vk\" (UniqueName: \"kubernetes.io/projected/789373aa-d138-4dff-a505-17410f8cf0c7-kube-api-access-sb6vk\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:51:49.618894 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:49.617617    4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/789373aa-d138-4dff-a505-17410f8cf0c7-maas-api-tls\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:51:49.618894 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:49.617633    4431 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/789373aa-d138-4dff-a505-17410f8cf0c7-metrics-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:51:49.732401 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:49.732319    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:51:49.869297 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:49.869264    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:51:50.260577 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:50.260546    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:51:50.319990 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:50.319961    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7940ea-565f5768b6-8nzcp"
Nov 03 22:51:50.320166 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:50.319999    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7940ea-565f5768b6-8nzcp" event={"ID":"789373aa-d138-4dff-a505-17410f8cf0c7","Type":"ContainerDied","Data":"e4bbf8bc8f2579f8c54adc3558267acbaaf9d9f39bb95c1d5c6f1a02eeae6f8f"}
Nov 03 22:51:50.320166 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:50.320069    4431 scope.go:122] "RemoveContainer" containerID="5e85065346e882020a2dca2a640cbad3a08b96ee865fcc60d3c6febf1c129745"
Nov 03 22:51:50.372330 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:50.372259    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7940ea-565f5768b6-8nzcp"]
Nov 03 22:51:50.384099 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:50.384071    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7940ea-565f5768b6-8nzcp"]
Nov 03 22:51:51.401782 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:51.401752    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:51:52.098628 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:52.098594    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="789373aa-d138-4dff-a505-17410f8cf0c7" path="/var/lib/kubelet/pods/789373aa-d138-4dff-a505-17410f8cf0c7/volumes"
Nov 03 22:51:52.193517 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:52.193481    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-9b841d-5c95644856-295pk"
Nov 03 22:51:52.917432 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:52.917400    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:51:53.985517 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:53.985484    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:51:54.494312 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:54.494280    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:51:54.595043 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:54.595006    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-50b9b5-7f48f98986-ddh6w" podUID="5f000cab-17f2-4a9f-a463-54983bba8566" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 22:51:55.955683 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:55.955653    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:51:56.624189 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:56.624159    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:51:56.773390 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:56.773360    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:51:56.983408 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:56.983372    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:51:57.738831 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:57.738797    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:51:59.443184 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:51:59.443126    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:52:01.118102 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:01.117888    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:52:01.260733 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:52:01.260698    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:52:02.969190 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:02.969088    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:52:04.105158 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:04.105130    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-50b9b5-7f48f98986-ddh6w"
Nov 03 22:52:04.202319 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:04.202287    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f000cab-17f2-4a9f-a463-54983bba8566-kube-api-access-jfl6v\" (UniqueName: \"kubernetes.io/projected/5f000cab-17f2-4a9f-a463-54983bba8566-kube-api-access-jfl6v\") pod \"5f000cab-17f2-4a9f-a463-54983bba8566\" (UID: \"5f000cab-17f2-4a9f-a463-54983bba8566\") "
Nov 03 22:52:04.202522 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:04.202489    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f000cab-17f2-4a9f-a463-54983bba8566-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f000cab-17f2-4a9f-a463-54983bba8566-plugins-config-volume\") pod \"5f000cab-17f2-4a9f-a463-54983bba8566\" (UID: \"5f000cab-17f2-4a9f-a463-54983bba8566\") "
Nov 03 22:52:04.203917 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:04.203619    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f000cab-17f2-4a9f-a463-54983bba8566-plugins-config-volume" pod "5f000cab-17f2-4a9f-a463-54983bba8566" (UID: "5f000cab-17f2-4a9f-a463-54983bba8566"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:52:04.206398 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:04.206367    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f000cab-17f2-4a9f-a463-54983bba8566-kube-api-access-jfl6v" pod "5f000cab-17f2-4a9f-a463-54983bba8566" (UID: "5f000cab-17f2-4a9f-a463-54983bba8566"). InnerVolumeSpecName "kube-api-access-jfl6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:52:04.304871 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:04.304833    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfl6v\" (UniqueName: \"kubernetes.io/projected/5f000cab-17f2-4a9f-a463-54983bba8566-kube-api-access-jfl6v\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:52:04.304871 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:04.304862    4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f000cab-17f2-4a9f-a463-54983bba8566-plugins-config-volume\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:52:04.429011 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:04.428978    4431 generic.go:356] "Generic (PLEG): container finished" podID="5f000cab-17f2-4a9f-a463-54983bba8566" containerID="fd1b2dc4dced9aba9d1592f9840982e16c9e7e4f336caf7d7cd01e2e23ad6d07" exitCode=0
Nov 03 22:52:04.429171 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:04.429029    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-50b9b5-7f48f98986-ddh6w"
Nov 03 22:52:04.429171 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:04.429024    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-50b9b5-7f48f98986-ddh6w" event={"ID":"5f000cab-17f2-4a9f-a463-54983bba8566","Type":"ContainerDied","Data":"fd1b2dc4dced9aba9d1592f9840982e16c9e7e4f336caf7d7cd01e2e23ad6d07"}
Nov 03 22:52:04.429171 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:04.429146    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-50b9b5-7f48f98986-ddh6w" event={"ID":"5f000cab-17f2-4a9f-a463-54983bba8566","Type":"ContainerDied","Data":"5bd843a3079c473d410a032efb75b6593e3d7067dd08c5c076890ad71e4e3fcd"}
Nov 03 22:52:04.429357 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:04.429173    4431 scope.go:122] "RemoveContainer" containerID="fd1b2dc4dced9aba9d1592f9840982e16c9e7e4f336caf7d7cd01e2e23ad6d07"
Nov 03 22:52:04.462426 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:04.458461    4431 scope.go:122] "RemoveContainer" containerID="fd1b2dc4dced9aba9d1592f9840982e16c9e7e4f336caf7d7cd01e2e23ad6d07"
Nov 03 22:52:04.462426 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:52:04.458866    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd1b2dc4dced9aba9d1592f9840982e16c9e7e4f336caf7d7cd01e2e23ad6d07\": container with ID starting with fd1b2dc4dced9aba9d1592f9840982e16c9e7e4f336caf7d7cd01e2e23ad6d07 not found: ID does not exist" containerID="fd1b2dc4dced9aba9d1592f9840982e16c9e7e4f336caf7d7cd01e2e23ad6d07"
Nov 03 22:52:04.462426 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:04.458904    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd1b2dc4dced9aba9d1592f9840982e16c9e7e4f336caf7d7cd01e2e23ad6d07"} err="failed to get container status \"fd1b2dc4dced9aba9d1592f9840982e16c9e7e4f336caf7d7cd01e2e23ad6d07\": rpc error: code = NotFound desc = could not find container \"fd1b2dc4dced9aba9d1592f9840982e16c9e7e4f336caf7d7cd01e2e23ad6d07\": container with ID starting with fd1b2dc4dced9aba9d1592f9840982e16c9e7e4f336caf7d7cd01e2e23ad6d07 not found: ID does not exist"
Nov 03 22:52:04.497308 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:04.497276    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-50b9b5-7f48f98986-ddh6w"]
Nov 03 22:52:04.503182 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:04.503157    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-50b9b5-7f48f98986-ddh6w"]
Nov 03 22:52:04.536028 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:04.535992    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:52:06.089678 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:06.089640    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f000cab-17f2-4a9f-a463-54983bba8566" path="/var/lib/kubelet/pods/5f000cab-17f2-4a9f-a463-54983bba8566/volumes"
Nov 03 22:52:06.137767 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:06.137733    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:52:07.715567 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:07.715535    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:52:07.984602 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:07.984519    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ccdb60-779c4b9f95-rrp8q"]
Nov 03 22:52:08.136623 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:08.132556    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:52:08.273811 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:08.273735    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:52:09.307504 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:09.307433    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:52:10.834708 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:10.834648    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:52:12.408563 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:12.408452    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:52:12.511030 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:12.510995    4431 generic.go:356] "Generic (PLEG): container finished" podID="db62e992-8f6d-4cba-b260-02bf3c06e839" containerID="86de258efe3b36a42c2aacc19c5e4e31ecf699a43467b762c39378d16a036b58" exitCode=0
Nov 03 22:52:12.511201 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:12.511069    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9" event={"ID":"db62e992-8f6d-4cba-b260-02bf3c06e839","Type":"ContainerDied","Data":"86de258efe3b36a42c2aacc19c5e4e31ecf699a43467b762c39378d16a036b58"}
Nov 03 22:52:12.511562 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:12.511549    4431 scope.go:122] "RemoveContainer" containerID="86de258efe3b36a42c2aacc19c5e4e31ecf699a43467b762c39378d16a036b58"
Nov 03 22:52:12.513332 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:12.513291    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:52:12.513427 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:12.513376    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:52:12.513427 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:12.513409    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:52:12.985130 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:12.985091    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ccdb60-779c4b9f95-rrp8q" podUID="52161b5e-fa32-4d5c-9695-b8fd60d6b87a" containerName="payload-pre-processing" containerID="cri-o://e420118dd5786a44b15110d21b74b96dc29dd1fd06e0f0f0d99af07b976b64eb" gracePeriod=25
Nov 03 22:52:13.519826 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:13.519797    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9" event={"ID":"db62e992-8f6d-4cba-b260-02bf3c06e839","Type":"ContainerStarted","Data":"1fae66ed8656bee092d94a35c3c71a2bd09f1c4ecd2da89ab85dc48bec420042"}
Nov 03 22:52:13.594842 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:13.594808    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9"
Nov 03 22:52:13.595063 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:13.595032    4431 patch_prober.go:29] interesting pod/e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Nov 03 22:52:13.595146 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:13.595102    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9" podUID="db62e992-8f6d-4cba-b260-02bf3c06e839" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Nov 03 22:52:13.787144 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:13.787064    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:52:14.278367 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:14.278328    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:52:14.425796 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:14.425761    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:52:14.595271 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:14.595181    4431 patch_prober.go:29] interesting pod/e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Nov 03 22:52:14.595637 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:14.595269    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9" podUID="db62e992-8f6d-4cba-b260-02bf3c06e839" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Nov 03 22:52:14.814873 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:14.814839    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:52:15.595642 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:15.595604    4431 patch_prober.go:29] interesting pod/e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Nov 03 22:52:15.596065 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:15.595657    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9" podUID="db62e992-8f6d-4cba-b260-02bf3c06e839" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Nov 03 22:52:16.001338 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:16.001283    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:52:16.590764 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:16.590726    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ccdb60-779c4b9f95-rrp8q" podUID="52161b5e-fa32-4d5c-9695-b8fd60d6b87a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 22:52:16.594962 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:16.594930    4431 patch_prober.go:29] interesting pod/e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Nov 03 22:52:16.595076 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:16.594979    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9" podUID="db62e992-8f6d-4cba-b260-02bf3c06e839" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Nov 03 22:52:17.582672 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:17.582631    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:52:17.595103 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:17.595065    4431 patch_prober.go:29] interesting pod/e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Nov 03 22:52:17.595430 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:17.595401    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9" podUID="db62e992-8f6d-4cba-b260-02bf3c06e839" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Nov 03 22:52:18.595119 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:18.595087    4431 patch_prober.go:29] interesting pod/e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Nov 03 22:52:18.595547 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:18.595139    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9" podUID="db62e992-8f6d-4cba-b260-02bf3c06e839" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Nov 03 22:52:19.068477 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:19.068441    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:52:19.206415 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:19.206377    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:52:19.601106 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:19.601073    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9"
Nov 03 22:52:19.601546 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:19.601493    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9"
Nov 03 22:52:19.602132 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:19.602112    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9"
Nov 03 22:52:20.826385 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:20.826349    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:52:21.844276 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:21.844194    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:52:22.028238 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:22.024938    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:52:22.091114 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:22.091083    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc"]
Nov 03 22:52:22.091398 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:22.091372    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc" podUID="89f34fbd-94ce-4fec-bc0c-38dab6758fa0" containerName="istio-proxy" containerID="cri-o://d2cfaaf94e5ebf5fd3ea00c7418cc3ab881248ea9c7205a4de3163f40108c7f4" gracePeriod=30
Nov 03 22:52:22.191195 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:22.191158    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:52:22.711266 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:22.711193    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:52:24.395522 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:24.395475    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:52:24.882181 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:24.882143    4431 patch_prober.go:29] interesting pod/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 22:52:24.882373 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:24.882250    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc" podUID="89f34fbd-94ce-4fec-bc0c-38dab6758fa0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 22:52:26.103027 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:26.102984    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:52:26.590774 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:26.590737    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ccdb60-779c4b9f95-rrp8q" podUID="52161b5e-fa32-4d5c-9695-b8fd60d6b87a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 22:52:27.373184 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:27.373122    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc"
Nov 03 22:52:27.429005 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:27.428974    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89f34fbd-94ce-4fec-bc0c-38dab6758fa0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89f34fbd-94ce-4fec-bc0c-38dab6758fa0-istio-data\") pod \"89f34fbd-94ce-4fec-bc0c-38dab6758fa0\" (UID: \"89f34fbd-94ce-4fec-bc0c-38dab6758fa0\") "
Nov 03 22:52:27.429190 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:27.429037    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89f34fbd-94ce-4fec-bc0c-38dab6758fa0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89f34fbd-94ce-4fec-bc0c-38dab6758fa0-istiod-ca-cert\") pod \"89f34fbd-94ce-4fec-bc0c-38dab6758fa0\" (UID: \"89f34fbd-94ce-4fec-bc0c-38dab6758fa0\") "
Nov 03 22:52:27.429190 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:27.429086    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89f34fbd-94ce-4fec-bc0c-38dab6758fa0-istio-token\" (UniqueName: \"kubernetes.io/projected/89f34fbd-94ce-4fec-bc0c-38dab6758fa0-istio-token\") pod \"89f34fbd-94ce-4fec-bc0c-38dab6758fa0\" (UID: \"89f34fbd-94ce-4fec-bc0c-38dab6758fa0\") "
Nov 03 22:52:27.429190 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:27.429129    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89f34fbd-94ce-4fec-bc0c-38dab6758fa0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89f34fbd-94ce-4fec-bc0c-38dab6758fa0-istio-envoy\") pod \"89f34fbd-94ce-4fec-bc0c-38dab6758fa0\" (UID: \"89f34fbd-94ce-4fec-bc0c-38dab6758fa0\") "
Nov 03 22:52:27.429190 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:27.429167    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89f34fbd-94ce-4fec-bc0c-38dab6758fa0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89f34fbd-94ce-4fec-bc0c-38dab6758fa0-credential-socket\") pod \"89f34fbd-94ce-4fec-bc0c-38dab6758fa0\" (UID: \"89f34fbd-94ce-4fec-bc0c-38dab6758fa0\") "
Nov 03 22:52:27.429477 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:27.429211    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/89f34fbd-94ce-4fec-bc0c-38dab6758fa0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89f34fbd-94ce-4fec-bc0c-38dab6758fa0-istio-podinfo\") pod \"89f34fbd-94ce-4fec-bc0c-38dab6758fa0\" (UID: \"89f34fbd-94ce-4fec-bc0c-38dab6758fa0\") "
Nov 03 22:52:27.429477 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:27.429304    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89f34fbd-94ce-4fec-bc0c-38dab6758fa0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89f34fbd-94ce-4fec-bc0c-38dab6758fa0-workload-certs\") pod \"89f34fbd-94ce-4fec-bc0c-38dab6758fa0\" (UID: \"89f34fbd-94ce-4fec-bc0c-38dab6758fa0\") "
Nov 03 22:52:27.429477 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:27.429407    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89f34fbd-94ce-4fec-bc0c-38dab6758fa0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89f34fbd-94ce-4fec-bc0c-38dab6758fa0-workload-socket\") pod \"89f34fbd-94ce-4fec-bc0c-38dab6758fa0\" (UID: \"89f34fbd-94ce-4fec-bc0c-38dab6758fa0\") "
Nov 03 22:52:27.429634 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:27.429605    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89f34fbd-94ce-4fec-bc0c-38dab6758fa0-kube-api-access-7qckt\" (UniqueName: \"kubernetes.io/projected/89f34fbd-94ce-4fec-bc0c-38dab6758fa0-kube-api-access-7qckt\") pod \"89f34fbd-94ce-4fec-bc0c-38dab6758fa0\" (UID: \"89f34fbd-94ce-4fec-bc0c-38dab6758fa0\") "
Nov 03 22:52:27.429803 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:27.429688    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89f34fbd-94ce-4fec-bc0c-38dab6758fa0-workload-certs" pod "89f34fbd-94ce-4fec-bc0c-38dab6758fa0" (UID: "89f34fbd-94ce-4fec-bc0c-38dab6758fa0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:52:27.429803 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:27.429759    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89f34fbd-94ce-4fec-bc0c-38dab6758fa0-istio-data" pod "89f34fbd-94ce-4fec-bc0c-38dab6758fa0" (UID: "89f34fbd-94ce-4fec-bc0c-38dab6758fa0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:52:27.429803 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:27.429778    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89f34fbd-94ce-4fec-bc0c-38dab6758fa0-credential-socket" pod "89f34fbd-94ce-4fec-bc0c-38dab6758fa0" (UID: "89f34fbd-94ce-4fec-bc0c-38dab6758fa0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:52:27.430057 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:27.429938    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89f34fbd-94ce-4fec-bc0c-38dab6758fa0-workload-socket" pod "89f34fbd-94ce-4fec-bc0c-38dab6758fa0" (UID: "89f34fbd-94ce-4fec-bc0c-38dab6758fa0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:52:27.430057 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:27.429964    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89f34fbd-94ce-4fec-bc0c-38dab6758fa0-istiod-ca-cert" pod "89f34fbd-94ce-4fec-bc0c-38dab6758fa0" (UID: "89f34fbd-94ce-4fec-bc0c-38dab6758fa0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:52:27.431332 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:27.430606    4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89f34fbd-94ce-4fec-bc0c-38dab6758fa0-workload-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:52:27.431332 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:27.430633    4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89f34fbd-94ce-4fec-bc0c-38dab6758fa0-istio-data\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:52:27.431332 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:27.430645    4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89f34fbd-94ce-4fec-bc0c-38dab6758fa0-istiod-ca-cert\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:52:27.431625 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:27.430660    4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89f34fbd-94ce-4fec-bc0c-38dab6758fa0-credential-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:52:27.431625 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:27.431604    4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89f34fbd-94ce-4fec-bc0c-38dab6758fa0-workload-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:52:27.432513 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:27.432482    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/89f34fbd-94ce-4fec-bc0c-38dab6758fa0-istio-podinfo" pod "89f34fbd-94ce-4fec-bc0c-38dab6758fa0" (UID: "89f34fbd-94ce-4fec-bc0c-38dab6758fa0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 22:52:27.432694 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:27.432672    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89f34fbd-94ce-4fec-bc0c-38dab6758fa0-istio-envoy" pod "89f34fbd-94ce-4fec-bc0c-38dab6758fa0" (UID: "89f34fbd-94ce-4fec-bc0c-38dab6758fa0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:52:27.432790 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:27.432771    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89f34fbd-94ce-4fec-bc0c-38dab6758fa0-kube-api-access-7qckt" pod "89f34fbd-94ce-4fec-bc0c-38dab6758fa0" (UID: "89f34fbd-94ce-4fec-bc0c-38dab6758fa0"). InnerVolumeSpecName "kube-api-access-7qckt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:52:27.434728 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:27.434701    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89f34fbd-94ce-4fec-bc0c-38dab6758fa0-istio-token" pod "89f34fbd-94ce-4fec-bc0c-38dab6758fa0" (UID: "89f34fbd-94ce-4fec-bc0c-38dab6758fa0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:52:27.532862 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:27.532829    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qckt\" (UniqueName: \"kubernetes.io/projected/89f34fbd-94ce-4fec-bc0c-38dab6758fa0-kube-api-access-7qckt\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:52:27.532862 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:27.532857    4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89f34fbd-94ce-4fec-bc0c-38dab6758fa0-istio-token\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:52:27.532862 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:27.532867    4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89f34fbd-94ce-4fec-bc0c-38dab6758fa0-istio-envoy\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:52:27.533101 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:27.532877    4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89f34fbd-94ce-4fec-bc0c-38dab6758fa0-istio-podinfo\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:52:27.629654 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:27.629574    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:52:27.657445 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:27.657412    4431 generic.go:356] "Generic (PLEG): container finished" podID="89f34fbd-94ce-4fec-bc0c-38dab6758fa0" containerID="d2cfaaf94e5ebf5fd3ea00c7418cc3ab881248ea9c7205a4de3163f40108c7f4" exitCode=0
Nov 03 22:52:27.657776 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:27.657464    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc"
Nov 03 22:52:27.657776 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:27.657482    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc" event={"ID":"89f34fbd-94ce-4fec-bc0c-38dab6758fa0","Type":"ContainerDied","Data":"d2cfaaf94e5ebf5fd3ea00c7418cc3ab881248ea9c7205a4de3163f40108c7f4"}
Nov 03 22:52:27.657776 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:27.657535    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc" event={"ID":"89f34fbd-94ce-4fec-bc0c-38dab6758fa0","Type":"ContainerDied","Data":"10410e2f0d8d7bccbf1861223ac1e45e26de5869268438d68ff7532b1557bad0"}
Nov 03 22:52:27.657776 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:27.657563    4431 scope.go:122] "RemoveContainer" containerID="d2cfaaf94e5ebf5fd3ea00c7418cc3ab881248ea9c7205a4de3163f40108c7f4"
Nov 03 22:52:27.669957 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:27.669939    4431 scope.go:122] "RemoveContainer" containerID="d2cfaaf94e5ebf5fd3ea00c7418cc3ab881248ea9c7205a4de3163f40108c7f4"
Nov 03 22:52:27.670316 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:52:27.670295    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2cfaaf94e5ebf5fd3ea00c7418cc3ab881248ea9c7205a4de3163f40108c7f4\": container with ID starting with d2cfaaf94e5ebf5fd3ea00c7418cc3ab881248ea9c7205a4de3163f40108c7f4 not found: ID does not exist" containerID="d2cfaaf94e5ebf5fd3ea00c7418cc3ab881248ea9c7205a4de3163f40108c7f4"
Nov 03 22:52:27.670395 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:27.670326    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2cfaaf94e5ebf5fd3ea00c7418cc3ab881248ea9c7205a4de3163f40108c7f4"} err="failed to get container status \"d2cfaaf94e5ebf5fd3ea00c7418cc3ab881248ea9c7205a4de3163f40108c7f4\": rpc error: code = NotFound desc = could not find container \"d2cfaaf94e5ebf5fd3ea00c7418cc3ab881248ea9c7205a4de3163f40108c7f4\": container with ID starting with d2cfaaf94e5ebf5fd3ea00c7418cc3ab881248ea9c7205a4de3163f40108c7f4 not found: ID does not exist"
Nov 03 22:52:27.711572 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:27.711541    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc"]
Nov 03 22:52:27.717732 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:27.717704    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-50b9b5-openshift-default-6f7ffb6f-jwjrc"]
Nov 03 22:52:28.089958 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:28.089927    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89f34fbd-94ce-4fec-bc0c-38dab6758fa0" path="/var/lib/kubelet/pods/89f34fbd-94ce-4fec-bc0c-38dab6758fa0/volumes"
Nov 03 22:52:29.167809 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:29.167775    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:52:30.732480 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:30.732445    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:52:31.260337 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:52:31.260299    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:52:32.371792 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:32.371756    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:52:33.065709 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:33.065677    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-fe7ca0-54c86d5ff5-xqxp7"]
Nov 03 22:52:33.199914 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:33.199880    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:52:33.318769 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:33.318742    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ccdb60-779c4b9f95-rrp8q"
Nov 03 22:52:33.347409 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:33.347373    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:52:33.419859 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:33.419829    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52161b5e-fa32-4d5c-9695-b8fd60d6b87a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52161b5e-fa32-4d5c-9695-b8fd60d6b87a-plugins-config-volume\") pod \"52161b5e-fa32-4d5c-9695-b8fd60d6b87a\" (UID: \"52161b5e-fa32-4d5c-9695-b8fd60d6b87a\") "
Nov 03 22:52:33.420303 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:33.419873    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52161b5e-fa32-4d5c-9695-b8fd60d6b87a-kube-api-access-2xqcq\" (UniqueName: \"kubernetes.io/projected/52161b5e-fa32-4d5c-9695-b8fd60d6b87a-kube-api-access-2xqcq\") pod \"52161b5e-fa32-4d5c-9695-b8fd60d6b87a\" (UID: \"52161b5e-fa32-4d5c-9695-b8fd60d6b87a\") "
Nov 03 22:52:33.420303 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:33.420191    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52161b5e-fa32-4d5c-9695-b8fd60d6b87a-plugins-config-volume" pod "52161b5e-fa32-4d5c-9695-b8fd60d6b87a" (UID: "52161b5e-fa32-4d5c-9695-b8fd60d6b87a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:52:33.420489 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:33.420473    4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52161b5e-fa32-4d5c-9695-b8fd60d6b87a-plugins-config-volume\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:52:33.422013 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:33.421994    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52161b5e-fa32-4d5c-9695-b8fd60d6b87a-kube-api-access-2xqcq" pod "52161b5e-fa32-4d5c-9695-b8fd60d6b87a" (UID: "52161b5e-fa32-4d5c-9695-b8fd60d6b87a"). InnerVolumeSpecName "kube-api-access-2xqcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:52:33.521849 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:33.521819    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xqcq\" (UniqueName: \"kubernetes.io/projected/52161b5e-fa32-4d5c-9695-b8fd60d6b87a-kube-api-access-2xqcq\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:52:33.735236 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:33.730666    4431 generic.go:356] "Generic (PLEG): container finished" podID="52161b5e-fa32-4d5c-9695-b8fd60d6b87a" containerID="e420118dd5786a44b15110d21b74b96dc29dd1fd06e0f0f0d99af07b976b64eb" exitCode=0
Nov 03 22:52:33.735236 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:33.730770    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ccdb60-779c4b9f95-rrp8q" event={"ID":"52161b5e-fa32-4d5c-9695-b8fd60d6b87a","Type":"ContainerDied","Data":"e420118dd5786a44b15110d21b74b96dc29dd1fd06e0f0f0d99af07b976b64eb"}
Nov 03 22:52:33.735236 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:33.730820    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ccdb60-779c4b9f95-rrp8q" event={"ID":"52161b5e-fa32-4d5c-9695-b8fd60d6b87a","Type":"ContainerDied","Data":"eadf4fed6949863c016c0b55bcfc85f0965328f6a91632de4a6d30cf6b28c7a0"}
Nov 03 22:52:33.735236 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:33.730847    4431 scope.go:122] "RemoveContainer" containerID="e420118dd5786a44b15110d21b74b96dc29dd1fd06e0f0f0d99af07b976b64eb"
Nov 03 22:52:33.735236 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:33.731003    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ccdb60-779c4b9f95-rrp8q"
Nov 03 22:52:33.759350 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:33.759329    4431 scope.go:122] "RemoveContainer" containerID="e420118dd5786a44b15110d21b74b96dc29dd1fd06e0f0f0d99af07b976b64eb"
Nov 03 22:52:33.759715 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:52:33.759690    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e420118dd5786a44b15110d21b74b96dc29dd1fd06e0f0f0d99af07b976b64eb\": container with ID starting with e420118dd5786a44b15110d21b74b96dc29dd1fd06e0f0f0d99af07b976b64eb not found: ID does not exist" containerID="e420118dd5786a44b15110d21b74b96dc29dd1fd06e0f0f0d99af07b976b64eb"
Nov 03 22:52:33.759857 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:33.759722    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e420118dd5786a44b15110d21b74b96dc29dd1fd06e0f0f0d99af07b976b64eb"} err="failed to get container status \"e420118dd5786a44b15110d21b74b96dc29dd1fd06e0f0f0d99af07b976b64eb\": rpc error: code = NotFound desc = could not find container \"e420118dd5786a44b15110d21b74b96dc29dd1fd06e0f0f0d99af07b976b64eb\": container with ID starting with e420118dd5786a44b15110d21b74b96dc29dd1fd06e0f0f0d99af07b976b64eb not found: ID does not exist"
Nov 03 22:52:33.795941 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:33.795906    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ccdb60-779c4b9f95-rrp8q"]
Nov 03 22:52:33.818423 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:33.818394    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ccdb60-779c4b9f95-rrp8q"]
Nov 03 22:52:33.964853 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:33.964800    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:52:34.090027 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:34.089951    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52161b5e-fa32-4d5c-9695-b8fd60d6b87a" path="/var/lib/kubelet/pods/52161b5e-fa32-4d5c-9695-b8fd60d6b87a/volumes"
Nov 03 22:52:35.476836 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:35.476806    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:52:36.761112 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:36.761068    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:52:37.271532 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:37.271495    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:52:37.422639 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:37.422605    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:52:37.568011 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:37.566925    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:52:37.828788 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:37.828711    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:52:38.066460 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:38.066419    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fe7ca0-54c86d5ff5-xqxp7" podUID="8a8675c0-7b21-40aa-bedd-217306c10ade" containerName="payload-pre-processing" containerID="cri-o://2cd494ca48a81e90c7c396807ff8e9fca37599e3afa776a819585eae82e0c685" gracePeriod=25
Nov 03 22:52:39.075896 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:39.075857    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:52:40.588662 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:40.588579    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:52:42.043731 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:42.043695    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fe7ca0-54c86d5ff5-xqxp7" podUID="8a8675c0-7b21-40aa-bedd-217306c10ade" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 22:52:42.121051 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:42.121016    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:52:43.592232 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:43.592191    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:52:43.979235 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:43.979184    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:52:44.121052 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:44.121018    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:52:44.381353 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:44.381268    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz"]
Nov 03 22:52:44.381520 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:44.381492    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz" podUID="79d7b9fc-6282-4aa1-bae1-e8cbe554eadf" containerName="istio-proxy" containerID="cri-o://5d95bdef12b7500251ab1fe06c11ced7f26d2cf546469eda91dc44b5df3a3bac" gracePeriod=30
Nov 03 22:52:44.392703 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:44.392677    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:52:45.268463 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:45.268426    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:52:46.806354 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:46.806320    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:52:48.298827 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:48.298786    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:52:49.022864 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:49.022826    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:52:49.679879 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:49.679853    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz"
Nov 03 22:52:49.769478 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:49.769444    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/79d7b9fc-6282-4aa1-bae1-e8cbe554eadf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79d7b9fc-6282-4aa1-bae1-e8cbe554eadf-istio-podinfo\") pod \"79d7b9fc-6282-4aa1-bae1-e8cbe554eadf\" (UID: \"79d7b9fc-6282-4aa1-bae1-e8cbe554eadf\") "
Nov 03 22:52:49.769676 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:49.769492    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79d7b9fc-6282-4aa1-bae1-e8cbe554eadf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79d7b9fc-6282-4aa1-bae1-e8cbe554eadf-workload-socket\") pod \"79d7b9fc-6282-4aa1-bae1-e8cbe554eadf\" (UID: \"79d7b9fc-6282-4aa1-bae1-e8cbe554eadf\") "
Nov 03 22:52:49.769676 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:49.769539    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79d7b9fc-6282-4aa1-bae1-e8cbe554eadf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79d7b9fc-6282-4aa1-bae1-e8cbe554eadf-istio-envoy\") pod \"79d7b9fc-6282-4aa1-bae1-e8cbe554eadf\" (UID: \"79d7b9fc-6282-4aa1-bae1-e8cbe554eadf\") "
Nov 03 22:52:49.769809 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:49.769676    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79d7b9fc-6282-4aa1-bae1-e8cbe554eadf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79d7b9fc-6282-4aa1-bae1-e8cbe554eadf-credential-socket\") pod \"79d7b9fc-6282-4aa1-bae1-e8cbe554eadf\" (UID: \"79d7b9fc-6282-4aa1-bae1-e8cbe554eadf\") "
Nov 03 22:52:49.769809 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:49.769788    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79d7b9fc-6282-4aa1-bae1-e8cbe554eadf-istio-token\" (UniqueName: \"kubernetes.io/projected/79d7b9fc-6282-4aa1-bae1-e8cbe554eadf-istio-token\") pod \"79d7b9fc-6282-4aa1-bae1-e8cbe554eadf\" (UID: \"79d7b9fc-6282-4aa1-bae1-e8cbe554eadf\") "
Nov 03 22:52:49.769909 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:49.769821    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79d7b9fc-6282-4aa1-bae1-e8cbe554eadf-workload-socket" pod "79d7b9fc-6282-4aa1-bae1-e8cbe554eadf" (UID: "79d7b9fc-6282-4aa1-bae1-e8cbe554eadf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:52:49.769909 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:49.769863    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79d7b9fc-6282-4aa1-bae1-e8cbe554eadf-kube-api-access-hxdqt\" (UniqueName: \"kubernetes.io/projected/79d7b9fc-6282-4aa1-bae1-e8cbe554eadf-kube-api-access-hxdqt\") pod \"79d7b9fc-6282-4aa1-bae1-e8cbe554eadf\" (UID: \"79d7b9fc-6282-4aa1-bae1-e8cbe554eadf\") "
Nov 03 22:52:49.770015 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:49.769938    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79d7b9fc-6282-4aa1-bae1-e8cbe554eadf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79d7b9fc-6282-4aa1-bae1-e8cbe554eadf-workload-certs\") pod \"79d7b9fc-6282-4aa1-bae1-e8cbe554eadf\" (UID: \"79d7b9fc-6282-4aa1-bae1-e8cbe554eadf\") "
Nov 03 22:52:49.770015 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:49.769995    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79d7b9fc-6282-4aa1-bae1-e8cbe554eadf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79d7b9fc-6282-4aa1-bae1-e8cbe554eadf-istio-data\") pod \"79d7b9fc-6282-4aa1-bae1-e8cbe554eadf\" (UID: \"79d7b9fc-6282-4aa1-bae1-e8cbe554eadf\") "
Nov 03 22:52:49.770129 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:49.770015    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79d7b9fc-6282-4aa1-bae1-e8cbe554eadf-credential-socket" pod "79d7b9fc-6282-4aa1-bae1-e8cbe554eadf" (UID: "79d7b9fc-6282-4aa1-bae1-e8cbe554eadf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:52:49.770188 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:49.770145    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79d7b9fc-6282-4aa1-bae1-e8cbe554eadf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79d7b9fc-6282-4aa1-bae1-e8cbe554eadf-istiod-ca-cert\") pod \"79d7b9fc-6282-4aa1-bae1-e8cbe554eadf\" (UID: \"79d7b9fc-6282-4aa1-bae1-e8cbe554eadf\") "
Nov 03 22:52:49.772371 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:49.771440    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79d7b9fc-6282-4aa1-bae1-e8cbe554eadf-workload-certs" pod "79d7b9fc-6282-4aa1-bae1-e8cbe554eadf" (UID: "79d7b9fc-6282-4aa1-bae1-e8cbe554eadf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:52:49.772371 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:49.772334    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79d7b9fc-6282-4aa1-bae1-e8cbe554eadf-istio-data" pod "79d7b9fc-6282-4aa1-bae1-e8cbe554eadf" (UID: "79d7b9fc-6282-4aa1-bae1-e8cbe554eadf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:52:49.772757 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:49.772733    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79d7b9fc-6282-4aa1-bae1-e8cbe554eadf-istio-token" pod "79d7b9fc-6282-4aa1-bae1-e8cbe554eadf" (UID: "79d7b9fc-6282-4aa1-bae1-e8cbe554eadf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:52:49.773844 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:49.773819    4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79d7b9fc-6282-4aa1-bae1-e8cbe554eadf-workload-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:52:49.774032 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:49.774006    4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79d7b9fc-6282-4aa1-bae1-e8cbe554eadf-credential-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:52:49.774032 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:49.774035    4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79d7b9fc-6282-4aa1-bae1-e8cbe554eadf-workload-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:52:49.779405 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:49.777364    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79d7b9fc-6282-4aa1-bae1-e8cbe554eadf-istiod-ca-cert" pod "79d7b9fc-6282-4aa1-bae1-e8cbe554eadf" (UID: "79d7b9fc-6282-4aa1-bae1-e8cbe554eadf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:52:49.794483 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:49.794450    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79d7b9fc-6282-4aa1-bae1-e8cbe554eadf-istio-envoy" pod "79d7b9fc-6282-4aa1-bae1-e8cbe554eadf" (UID: "79d7b9fc-6282-4aa1-bae1-e8cbe554eadf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:52:49.794607 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:49.794506    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79d7b9fc-6282-4aa1-bae1-e8cbe554eadf-kube-api-access-hxdqt" pod "79d7b9fc-6282-4aa1-bae1-e8cbe554eadf" (UID: "79d7b9fc-6282-4aa1-bae1-e8cbe554eadf"). InnerVolumeSpecName "kube-api-access-hxdqt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:52:49.796623 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:49.796590    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/79d7b9fc-6282-4aa1-bae1-e8cbe554eadf-istio-podinfo" pod "79d7b9fc-6282-4aa1-bae1-e8cbe554eadf" (UID: "79d7b9fc-6282-4aa1-bae1-e8cbe554eadf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 22:52:49.829932 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:49.829904    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:52:49.866829 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:49.866791    4431 generic.go:356] "Generic (PLEG): container finished" podID="79d7b9fc-6282-4aa1-bae1-e8cbe554eadf" containerID="5d95bdef12b7500251ab1fe06c11ced7f26d2cf546469eda91dc44b5df3a3bac" exitCode=0
Nov 03 22:52:49.866998 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:49.866839    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz" event={"ID":"79d7b9fc-6282-4aa1-bae1-e8cbe554eadf","Type":"ContainerDied","Data":"5d95bdef12b7500251ab1fe06c11ced7f26d2cf546469eda91dc44b5df3a3bac"}
Nov 03 22:52:49.866998 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:49.866867    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz"
Nov 03 22:52:49.866998 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:49.866923    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz" event={"ID":"79d7b9fc-6282-4aa1-bae1-e8cbe554eadf","Type":"ContainerDied","Data":"b3cee509e6e8fcd7ec4963516e6ecdecb824606ea1055bb05cb40e6a5d2ddbae"}
Nov 03 22:52:49.866998 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:49.866956    4431 scope.go:122] "RemoveContainer" containerID="5d95bdef12b7500251ab1fe06c11ced7f26d2cf546469eda91dc44b5df3a3bac"
Nov 03 22:52:49.875611 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:49.875583    4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79d7b9fc-6282-4aa1-bae1-e8cbe554eadf-istio-podinfo\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:52:49.875611 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:49.875613    4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79d7b9fc-6282-4aa1-bae1-e8cbe554eadf-istio-envoy\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:52:49.875768 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:49.875626    4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79d7b9fc-6282-4aa1-bae1-e8cbe554eadf-istio-token\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:52:49.875768 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:49.875637    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxdqt\" (UniqueName: \"kubernetes.io/projected/79d7b9fc-6282-4aa1-bae1-e8cbe554eadf-kube-api-access-hxdqt\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:52:49.875768 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:49.875652    4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79d7b9fc-6282-4aa1-bae1-e8cbe554eadf-istio-data\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:52:49.875768 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:49.875665    4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79d7b9fc-6282-4aa1-bae1-e8cbe554eadf-istiod-ca-cert\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:52:49.878946 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:49.878919    4431 scope.go:122] "RemoveContainer" containerID="cf8ef89b306aecedf3c8c429d71f4a272f85645abec9c84d410be06c528f3acd"
Nov 03 22:52:49.892732 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:49.892696    4431 scope.go:122] "RemoveContainer" containerID="5d95bdef12b7500251ab1fe06c11ced7f26d2cf546469eda91dc44b5df3a3bac"
Nov 03 22:52:49.893100 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:52:49.893077    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d95bdef12b7500251ab1fe06c11ced7f26d2cf546469eda91dc44b5df3a3bac\": container with ID starting with 5d95bdef12b7500251ab1fe06c11ced7f26d2cf546469eda91dc44b5df3a3bac not found: ID does not exist" containerID="5d95bdef12b7500251ab1fe06c11ced7f26d2cf546469eda91dc44b5df3a3bac"
Nov 03 22:52:49.893196 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:49.893114    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d95bdef12b7500251ab1fe06c11ced7f26d2cf546469eda91dc44b5df3a3bac"} err="failed to get container status \"5d95bdef12b7500251ab1fe06c11ced7f26d2cf546469eda91dc44b5df3a3bac\": rpc error: code = NotFound desc = could not find container \"5d95bdef12b7500251ab1fe06c11ced7f26d2cf546469eda91dc44b5df3a3bac\": container with ID starting with 5d95bdef12b7500251ab1fe06c11ced7f26d2cf546469eda91dc44b5df3a3bac not found: ID does not exist"
Nov 03 22:52:49.893196 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:49.893140    4431 scope.go:122] "RemoveContainer" containerID="cf8ef89b306aecedf3c8c429d71f4a272f85645abec9c84d410be06c528f3acd"
Nov 03 22:52:49.893466 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:52:49.893433    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf8ef89b306aecedf3c8c429d71f4a272f85645abec9c84d410be06c528f3acd\": container with ID starting with cf8ef89b306aecedf3c8c429d71f4a272f85645abec9c84d410be06c528f3acd not found: ID does not exist" containerID="cf8ef89b306aecedf3c8c429d71f4a272f85645abec9c84d410be06c528f3acd"
Nov 03 22:52:49.893578 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:49.893474    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf8ef89b306aecedf3c8c429d71f4a272f85645abec9c84d410be06c528f3acd"} err="failed to get container status \"cf8ef89b306aecedf3c8c429d71f4a272f85645abec9c84d410be06c528f3acd\": rpc error: code = NotFound desc = could not find container \"cf8ef89b306aecedf3c8c429d71f4a272f85645abec9c84d410be06c528f3acd\": container with ID starting with cf8ef89b306aecedf3c8c429d71f4a272f85645abec9c84d410be06c528f3acd not found: ID does not exist"
Nov 03 22:52:49.920267 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:49.920240    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz"]
Nov 03 22:52:49.926106 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:49.926082    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-ccdb60-openshift-default-678d965f7c-cttbz"]
Nov 03 22:52:50.089607 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:50.089575    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79d7b9fc-6282-4aa1-bae1-e8cbe554eadf" path="/var/lib/kubelet/pods/79d7b9fc-6282-4aa1-bae1-e8cbe554eadf/volumes"
Nov 03 22:52:51.483714 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:51.483682    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:52:52.043490 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:52.043442    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fe7ca0-54c86d5ff5-xqxp7" podUID="8a8675c0-7b21-40aa-bedd-217306c10ade" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 22:52:53.245330 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:53.245279    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:52:54.146301 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:54.146268    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:52:54.919630 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:54.919592    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:52:56.600044 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:56.600007    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:52:56.755656 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:56.755618    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:52:56.814475 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:56.814448    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mx5q8_ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea/ovn-acl-logging/0.log"
Nov 03 22:52:56.824877 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:56.824854    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mx5q8_ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea/ovn-acl-logging/0.log"
Nov 03 22:52:56.891851 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:56.891766    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:52:58.113689 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:58.113660    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:52:58.350123 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:58.350098    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fe7ca0-54c86d5ff5-xqxp7"
Nov 03 22:52:58.434316 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:58.434280    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a8675c0-7b21-40aa-bedd-217306c10ade-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a8675c0-7b21-40aa-bedd-217306c10ade-plugins-config-volume\") pod \"8a8675c0-7b21-40aa-bedd-217306c10ade\" (UID: \"8a8675c0-7b21-40aa-bedd-217306c10ade\") "
Nov 03 22:52:58.434509 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:58.434437    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a8675c0-7b21-40aa-bedd-217306c10ade-kube-api-access-k5nhz\" (UniqueName: \"kubernetes.io/projected/8a8675c0-7b21-40aa-bedd-217306c10ade-kube-api-access-k5nhz\") pod \"8a8675c0-7b21-40aa-bedd-217306c10ade\" (UID: \"8a8675c0-7b21-40aa-bedd-217306c10ade\") "
Nov 03 22:52:58.434719 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:58.434693    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a8675c0-7b21-40aa-bedd-217306c10ade-plugins-config-volume" pod "8a8675c0-7b21-40aa-bedd-217306c10ade" (UID: "8a8675c0-7b21-40aa-bedd-217306c10ade"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:52:58.434913 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:58.434896    4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a8675c0-7b21-40aa-bedd-217306c10ade-plugins-config-volume\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:52:58.436519 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:58.436496    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a8675c0-7b21-40aa-bedd-217306c10ade-kube-api-access-k5nhz" pod "8a8675c0-7b21-40aa-bedd-217306c10ade" (UID: "8a8675c0-7b21-40aa-bedd-217306c10ade"). InnerVolumeSpecName "kube-api-access-k5nhz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:52:58.536247 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:58.536191    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5nhz\" (UniqueName: \"kubernetes.io/projected/8a8675c0-7b21-40aa-bedd-217306c10ade-kube-api-access-k5nhz\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:52:58.738484 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:58.738452    4431 scope.go:122] "RemoveContainer" containerID="2cd494ca48a81e90c7c396807ff8e9fca37599e3afa776a819585eae82e0c685"
Nov 03 22:52:58.935116 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:58.935075    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fe7ca0-54c86d5ff5-xqxp7" event={"ID":"8a8675c0-7b21-40aa-bedd-217306c10ade","Type":"ContainerDied","Data":"2cd494ca48a81e90c7c396807ff8e9fca37599e3afa776a819585eae82e0c685"}
Nov 03 22:52:58.935296 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:58.935168    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fe7ca0-54c86d5ff5-xqxp7" event={"ID":"8a8675c0-7b21-40aa-bedd-217306c10ade","Type":"ContainerDied","Data":"3d2bb6da7d693e96519672cb4a71a7c4bf62595fcd48c17c82e8b82aa1f871a1"}
Nov 03 22:52:58.935296 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:58.935083    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fe7ca0-54c86d5ff5-xqxp7"
Nov 03 22:52:59.012173 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:59.012090    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-fe7ca0-54c86d5ff5-xqxp7"]
Nov 03 22:52:59.016563 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:59.016533    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-fe7ca0-54c86d5ff5-xqxp7"]
Nov 03 22:52:59.801319 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:52:59.801278    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:53:00.089457 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:00.089377    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a8675c0-7b21-40aa-bedd-217306c10ade" path="/var/lib/kubelet/pods/8a8675c0-7b21-40aa-bedd-217306c10ade/volumes"
Nov 03 22:53:01.260645 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:53:01.260606    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:53:01.303013 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:01.302980    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:53:03.010034 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:03.009932    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:53:04.271841 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:04.271774    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:53:04.569432 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:04.569326    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:53:04.728183 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:04.728149    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:53:04.873884 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:04.873802    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:53:05.344257 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:05.344201    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:53:06.621154 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:06.621069    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:53:08.203853 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:08.203821    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:53:08.395515 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:08.395482    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:53:09.698914 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:09.698873    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:53:11.429481 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:11.429440    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:53:11.657663 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:11.657631    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:53:11.798689 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:11.798604    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:53:12.005772 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:12.005738    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:53:13.213181 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:13.213146    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:53:14.755901 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:14.755852    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:53:15.976380 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:15.976338    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:53:16.478340 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:16.478304    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:53:18.099333 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:18.099296    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:53:19.704689 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:19.704657    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:53:20.540005 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:20.539960    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w5-b023a2/auth-test-model-b023a2-kserve-7f85457dc7-pvx2r"]
Nov 03 22:53:20.540845 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:20.540808    4431 cpu_manager.go:414] "containerMap: removing container" podUID="8a8675c0-7b21-40aa-bedd-217306c10ade" containerName="payload-pre-processing"
Nov 03 22:53:20.540845 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:20.540834    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a8675c0-7b21-40aa-bedd-217306c10ade" containerName="payload-pre-processing"
Nov 03 22:53:20.541067 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:20.540855    4431 cpu_manager.go:414] "containerMap: removing container" podUID="789373aa-d138-4dff-a505-17410f8cf0c7" containerName="maas-api"
Nov 03 22:53:20.541067 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:20.540860    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="789373aa-d138-4dff-a505-17410f8cf0c7" containerName="maas-api"
Nov 03 22:53:20.541067 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:20.540869    4431 cpu_manager.go:414] "containerMap: removing container" podUID="79d7b9fc-6282-4aa1-bae1-e8cbe554eadf" containerName="istio-proxy"
Nov 03 22:53:20.541067 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:20.540875    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79d7b9fc-6282-4aa1-bae1-e8cbe554eadf" containerName="istio-proxy"
Nov 03 22:53:20.541067 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:20.540887    4431 cpu_manager.go:414] "containerMap: removing container" podUID="4f04425d-fb46-454f-84af-dd29b0208c21" containerName="main"
Nov 03 22:53:20.541067 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:20.540893    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f04425d-fb46-454f-84af-dd29b0208c21" containerName="main"
Nov 03 22:53:20.541067 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:20.540934    4431 cpu_manager.go:414] "containerMap: removing container" podUID="52161b5e-fa32-4d5c-9695-b8fd60d6b87a" containerName="payload-pre-processing"
Nov 03 22:53:20.541067 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:20.540942    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52161b5e-fa32-4d5c-9695-b8fd60d6b87a" containerName="payload-pre-processing"
Nov 03 22:53:20.541067 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:20.540959    4431 cpu_manager.go:414] "containerMap: removing container" podUID="5f000cab-17f2-4a9f-a463-54983bba8566" containerName="payload-processing"
Nov 03 22:53:20.541067 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:20.540967    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f000cab-17f2-4a9f-a463-54983bba8566" containerName="payload-processing"
Nov 03 22:53:20.541067 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:20.540981    4431 cpu_manager.go:414] "containerMap: removing container" podUID="89f34fbd-94ce-4fec-bc0c-38dab6758fa0" containerName="istio-proxy"
Nov 03 22:53:20.541067 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:20.540989    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89f34fbd-94ce-4fec-bc0c-38dab6758fa0" containerName="istio-proxy"
Nov 03 22:53:20.541067 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:20.541004    4431 cpu_manager.go:414] "containerMap: removing container" podUID="79d7b9fc-6282-4aa1-bae1-e8cbe554eadf" containerName="istio-proxy"
Nov 03 22:53:20.541067 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:20.541011    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79d7b9fc-6282-4aa1-bae1-e8cbe554eadf" containerName="istio-proxy"
Nov 03 22:53:20.541794 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:20.541776    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f04425d-fb46-454f-84af-dd29b0208c21" containerName="main"
Nov 03 22:53:20.541846 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:20.541804    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="52161b5e-fa32-4d5c-9695-b8fd60d6b87a" containerName="payload-pre-processing"
Nov 03 22:53:20.541846 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:20.541812    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f000cab-17f2-4a9f-a463-54983bba8566" containerName="payload-processing"
Nov 03 22:53:20.541846 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:20.541821    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="89f34fbd-94ce-4fec-bc0c-38dab6758fa0" containerName="istio-proxy"
Nov 03 22:53:20.541846 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:20.541830    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="79d7b9fc-6282-4aa1-bae1-e8cbe554eadf" containerName="istio-proxy"
Nov 03 22:53:20.541846 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:20.541840    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a8675c0-7b21-40aa-bedd-217306c10ade" containerName="payload-pre-processing"
Nov 03 22:53:20.541846 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:20.541846    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="789373aa-d138-4dff-a505-17410f8cf0c7" containerName="maas-api"
Nov 03 22:53:20.542079 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:20.541852    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="79d7b9fc-6282-4aa1-bae1-e8cbe554eadf" containerName="istio-proxy"
Nov 03 22:53:20.547403 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:20.547383    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-b023a2/auth-test-model-b023a2-kserve-7f85457dc7-pvx2r"
Nov 03 22:53:20.566013 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:20.565978    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-b023a2/auth-test-model-b023a2-kserve-7f85457dc7-pvx2r"]
Nov 03 22:53:20.571988 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:20.571856    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-b023a2\"/\"openshift-service-ca.crt\""
Nov 03 22:53:20.571988 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:20.571906    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-b023a2\"/\"auth-test-model-b023a2-kserve-self-signed-certs\""
Nov 03 22:53:20.571988 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:20.571958    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-b023a2\"/\"kube-root-ca.crt\""
Nov 03 22:53:20.573768 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:20.573742    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-b023a2\"/\"default-dockercfg-s9rcn\""
Nov 03 22:53:20.636257 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:20.636169    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:53:20.708014 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:20.707977    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ba07c9c-1005-4d6f-b0af-ac5463bf588f-model-cache\") pod \"auth-test-model-b023a2-kserve-7f85457dc7-pvx2r\" (UID: \"1ba07c9c-1005-4d6f-b0af-ac5463bf588f\") " pod="ai-tenant-e2e-shared-a-w5-b023a2/auth-test-model-b023a2-kserve-7f85457dc7-pvx2r"
Nov 03 22:53:20.708435 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:20.708166    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ba07c9c-1005-4d6f-b0af-ac5463bf588f-tls-certs\") pod \"auth-test-model-b023a2-kserve-7f85457dc7-pvx2r\" (UID: \"1ba07c9c-1005-4d6f-b0af-ac5463bf588f\") " pod="ai-tenant-e2e-shared-a-w5-b023a2/auth-test-model-b023a2-kserve-7f85457dc7-pvx2r"
Nov 03 22:53:20.708435 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:20.708325    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ba07c9c-1005-4d6f-b0af-ac5463bf588f-tmp-dir\") pod \"auth-test-model-b023a2-kserve-7f85457dc7-pvx2r\" (UID: \"1ba07c9c-1005-4d6f-b0af-ac5463bf588f\") " pod="ai-tenant-e2e-shared-a-w5-b023a2/auth-test-model-b023a2-kserve-7f85457dc7-pvx2r"
Nov 03 22:53:20.708435 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:20.708384    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ba07c9c-1005-4d6f-b0af-ac5463bf588f-dshm\") pod \"auth-test-model-b023a2-kserve-7f85457dc7-pvx2r\" (UID: \"1ba07c9c-1005-4d6f-b0af-ac5463bf588f\") " pod="ai-tenant-e2e-shared-a-w5-b023a2/auth-test-model-b023a2-kserve-7f85457dc7-pvx2r"
Nov 03 22:53:20.708567 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:20.708477    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ba07c9c-1005-4d6f-b0af-ac5463bf588f-home\") pod \"auth-test-model-b023a2-kserve-7f85457dc7-pvx2r\" (UID: \"1ba07c9c-1005-4d6f-b0af-ac5463bf588f\") " pod="ai-tenant-e2e-shared-a-w5-b023a2/auth-test-model-b023a2-kserve-7f85457dc7-pvx2r"
Nov 03 22:53:20.708567 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:20.708497    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzs6f\" (UniqueName: \"kubernetes.io/projected/1ba07c9c-1005-4d6f-b0af-ac5463bf588f-kube-api-access-tzs6f\") pod \"auth-test-model-b023a2-kserve-7f85457dc7-pvx2r\" (UID: \"1ba07c9c-1005-4d6f-b0af-ac5463bf588f\") " pod="ai-tenant-e2e-shared-a-w5-b023a2/auth-test-model-b023a2-kserve-7f85457dc7-pvx2r"
Nov 03 22:53:20.818283 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:20.813095    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ba07c9c-1005-4d6f-b0af-ac5463bf588f-home\") pod \"auth-test-model-b023a2-kserve-7f85457dc7-pvx2r\" (UID: \"1ba07c9c-1005-4d6f-b0af-ac5463bf588f\") " pod="ai-tenant-e2e-shared-a-w5-b023a2/auth-test-model-b023a2-kserve-7f85457dc7-pvx2r"
Nov 03 22:53:20.818283 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:20.813153    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzs6f\" (UniqueName: \"kubernetes.io/projected/1ba07c9c-1005-4d6f-b0af-ac5463bf588f-kube-api-access-tzs6f\") pod \"auth-test-model-b023a2-kserve-7f85457dc7-pvx2r\" (UID: \"1ba07c9c-1005-4d6f-b0af-ac5463bf588f\") " pod="ai-tenant-e2e-shared-a-w5-b023a2/auth-test-model-b023a2-kserve-7f85457dc7-pvx2r"
Nov 03 22:53:20.818283 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:20.813274    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ba07c9c-1005-4d6f-b0af-ac5463bf588f-model-cache\") pod \"auth-test-model-b023a2-kserve-7f85457dc7-pvx2r\" (UID: \"1ba07c9c-1005-4d6f-b0af-ac5463bf588f\") " pod="ai-tenant-e2e-shared-a-w5-b023a2/auth-test-model-b023a2-kserve-7f85457dc7-pvx2r"
Nov 03 22:53:20.818283 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:20.813398    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ba07c9c-1005-4d6f-b0af-ac5463bf588f-tls-certs\") pod \"auth-test-model-b023a2-kserve-7f85457dc7-pvx2r\" (UID: \"1ba07c9c-1005-4d6f-b0af-ac5463bf588f\") " pod="ai-tenant-e2e-shared-a-w5-b023a2/auth-test-model-b023a2-kserve-7f85457dc7-pvx2r"
Nov 03 22:53:20.818283 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:20.813505    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ba07c9c-1005-4d6f-b0af-ac5463bf588f-tmp-dir\") pod \"auth-test-model-b023a2-kserve-7f85457dc7-pvx2r\" (UID: \"1ba07c9c-1005-4d6f-b0af-ac5463bf588f\") " pod="ai-tenant-e2e-shared-a-w5-b023a2/auth-test-model-b023a2-kserve-7f85457dc7-pvx2r"
Nov 03 22:53:20.818283 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:20.813532    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ba07c9c-1005-4d6f-b0af-ac5463bf588f-dshm\") pod \"auth-test-model-b023a2-kserve-7f85457dc7-pvx2r\" (UID: \"1ba07c9c-1005-4d6f-b0af-ac5463bf588f\") " pod="ai-tenant-e2e-shared-a-w5-b023a2/auth-test-model-b023a2-kserve-7f85457dc7-pvx2r"
Nov 03 22:53:20.818283 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:20.814366    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ba07c9c-1005-4d6f-b0af-ac5463bf588f-model-cache\") pod \"auth-test-model-b023a2-kserve-7f85457dc7-pvx2r\" (UID: \"1ba07c9c-1005-4d6f-b0af-ac5463bf588f\") " pod="ai-tenant-e2e-shared-a-w5-b023a2/auth-test-model-b023a2-kserve-7f85457dc7-pvx2r"
Nov 03 22:53:20.818283 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:20.814618    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ba07c9c-1005-4d6f-b0af-ac5463bf588f-home\") pod \"auth-test-model-b023a2-kserve-7f85457dc7-pvx2r\" (UID: \"1ba07c9c-1005-4d6f-b0af-ac5463bf588f\") " pod="ai-tenant-e2e-shared-a-w5-b023a2/auth-test-model-b023a2-kserve-7f85457dc7-pvx2r"
Nov 03 22:53:20.818283 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:20.815168    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ba07c9c-1005-4d6f-b0af-ac5463bf588f-tmp-dir\") pod \"auth-test-model-b023a2-kserve-7f85457dc7-pvx2r\" (UID: \"1ba07c9c-1005-4d6f-b0af-ac5463bf588f\") " pod="ai-tenant-e2e-shared-a-w5-b023a2/auth-test-model-b023a2-kserve-7f85457dc7-pvx2r"
Nov 03 22:53:20.832997 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:20.832934    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ba07c9c-1005-4d6f-b0af-ac5463bf588f-tls-certs\") pod \"auth-test-model-b023a2-kserve-7f85457dc7-pvx2r\" (UID: \"1ba07c9c-1005-4d6f-b0af-ac5463bf588f\") " pod="ai-tenant-e2e-shared-a-w5-b023a2/auth-test-model-b023a2-kserve-7f85457dc7-pvx2r"
Nov 03 22:53:20.833743 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:20.833716    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ba07c9c-1005-4d6f-b0af-ac5463bf588f-dshm\") pod \"auth-test-model-b023a2-kserve-7f85457dc7-pvx2r\" (UID: \"1ba07c9c-1005-4d6f-b0af-ac5463bf588f\") " pod="ai-tenant-e2e-shared-a-w5-b023a2/auth-test-model-b023a2-kserve-7f85457dc7-pvx2r"
Nov 03 22:53:20.855964 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:20.855924    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzs6f\" (UniqueName: \"kubernetes.io/projected/1ba07c9c-1005-4d6f-b0af-ac5463bf588f-kube-api-access-tzs6f\") pod \"auth-test-model-b023a2-kserve-7f85457dc7-pvx2r\" (UID: \"1ba07c9c-1005-4d6f-b0af-ac5463bf588f\") " pod="ai-tenant-e2e-shared-a-w5-b023a2/auth-test-model-b023a2-kserve-7f85457dc7-pvx2r"
Nov 03 22:53:20.864778 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:20.863646    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-b023a2/auth-test-model-b023a2-kserve-7f85457dc7-pvx2r"
Nov 03 22:53:21.184887 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:21.184862    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-b023a2/auth-test-model-b023a2-kserve-7f85457dc7-pvx2r"]
Nov 03 22:53:21.187271 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:53:21.187245    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1ba07c9c_1005_4d6f_b0af_ac5463bf588f.slice/crio-92c5fdafeccdad0842116903fff7d0f6da3790a1961e5e90d30d186ec088ed61 WatchSource:0}: Error finding container 92c5fdafeccdad0842116903fff7d0f6da3790a1961e5e90d30d186ec088ed61: Status 404 returned error can't find the container with id 92c5fdafeccdad0842116903fff7d0f6da3790a1961e5e90d30d186ec088ed61
Nov 03 22:53:21.235406 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:21.235374    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:53:22.144940 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:22.144901    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-b023a2/auth-test-model-b023a2-kserve-7f85457dc7-pvx2r" event={"ID":"1ba07c9c-1005-4d6f-b0af-ac5463bf588f","Type":"ContainerStarted","Data":"6aee4ccf41d59433d93452cd39b692cfd9ac057e2d808bed313f7d05f81e789a"}
Nov 03 22:53:22.145413 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:22.144951    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-b023a2/auth-test-model-b023a2-kserve-7f85457dc7-pvx2r" event={"ID":"1ba07c9c-1005-4d6f-b0af-ac5463bf588f","Type":"ContainerStarted","Data":"92c5fdafeccdad0842116903fff7d0f6da3790a1961e5e90d30d186ec088ed61"}
Nov 03 22:53:22.189432 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:22.189373    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w5-b023a2/auth-test-model-b023a2-kserve-7f85457dc7-pvx2r" podStartSLOduration=2.18935849 podStartE2EDuration="2.18935849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:53:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:53:22.185865392 +0000 UTC m=+2426.211721256" watchObservedRunningTime="2026-11-03 22:53:22.18935849 +0000 UTC m=+2426.215214352"
Nov 03 22:53:22.864829 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:22.864792    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w5-b023a2/auth-test-model-b023a2-kserve-7f85457dc7-pvx2r"
Nov 03 22:53:22.899649 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:22.899616    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:53:24.685913 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:24.685880    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:53:26.421544 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:26.421490    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:53:26.673779 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:26.673690    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-341c51-857b566498-5d2tg"]
Nov 03 22:53:26.704317 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:26.704283    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-341c51-76d98b8f6c-cqm9p"]
Nov 03 22:53:26.856899 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:26.856865    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:53:27.003959 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:27.003859    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:53:28.081398 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:28.081367    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:53:29.506065 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:29.506029    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:53:30.864934 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:30.864906    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w5-b023a2/auth-test-model-b023a2-kserve-7f85457dc7-pvx2r"
Nov 03 22:53:30.877456 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:30.877429    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w5-b023a2/auth-test-model-b023a2-kserve-7f85457dc7-pvx2r"
Nov 03 22:53:31.055077 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:31.055043    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:53:31.215385 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:31.215359    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w5-b023a2/auth-test-model-b023a2-kserve-7f85457dc7-pvx2r"
Nov 03 22:53:31.261282 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:53:31.261247    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:53:31.674446 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:31.674405    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-341c51-857b566498-5d2tg" podUID="97ea71c9-692d-49fb-a079-183bda697421" containerName="payload-processing" containerID="cri-o://f285c04e5ad7f1b0e10d0bf65034d1717b76113e91a72f5b07a7fa8804242eec" gracePeriod=25
Nov 03 22:53:31.705372 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:31.705334    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-341c51-76d98b8f6c-cqm9p" podUID="16008341-844a-48a1-80dd-9f804fbcae4a" containerName="payload-pre-processing" containerID="cri-o://409774e8ed984be083cbd48927211d23ba434226afca97ce4858a73fa9c8003e" gracePeriod=25
Nov 03 22:53:32.205383 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:32.205342    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-341c51-857b566498-5d2tg" podUID="97ea71c9-692d-49fb-a079-183bda697421" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 22:53:32.205794 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:32.205354    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-341c51-76d98b8f6c-cqm9p" podUID="16008341-844a-48a1-80dd-9f804fbcae4a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 22:53:32.722448 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:32.722412    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:53:34.211325 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:34.211290    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:53:35.808432 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:35.808397    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:53:37.323188 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:37.323156    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:53:38.903118 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:38.903084    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:53:40.527666 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:40.527631    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:53:42.085431 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:42.085393    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:53:42.205394 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:42.205355    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-341c51-76d98b8f6c-cqm9p" podUID="16008341-844a-48a1-80dd-9f804fbcae4a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 22:53:42.205738 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:42.205368    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-341c51-857b566498-5d2tg" podUID="97ea71c9-692d-49fb-a079-183bda697421" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 22:53:43.578358 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:43.578324    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:53:43.718174 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:43.718138    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:53:45.361327 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:45.361290    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:53:47.114918 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:47.114883    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:53:48.690004 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:48.689966    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:53:49.134115 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:49.134032    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:53:50.321043 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:50.321013    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:53:51.778073 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:51.778040    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:53:51.992666 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:51.992639    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-341c51-857b566498-5d2tg"
Nov 03 22:53:52.026169 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:52.026145    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-341c51-76d98b8f6c-cqm9p"
Nov 03 22:53:52.072007 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:52.071970    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97ea71c9-692d-49fb-a079-183bda697421-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97ea71c9-692d-49fb-a079-183bda697421-plugins-config-volume\") pod \"97ea71c9-692d-49fb-a079-183bda697421\" (UID: \"97ea71c9-692d-49fb-a079-183bda697421\") "
Nov 03 22:53:52.072341 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:52.072313    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97ea71c9-692d-49fb-a079-183bda697421-kube-api-access-xkwgv\" (UniqueName: \"kubernetes.io/projected/97ea71c9-692d-49fb-a079-183bda697421-kube-api-access-xkwgv\") pod \"97ea71c9-692d-49fb-a079-183bda697421\" (UID: \"97ea71c9-692d-49fb-a079-183bda697421\") "
Nov 03 22:53:52.072695 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:52.072667    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97ea71c9-692d-49fb-a079-183bda697421-plugins-config-volume" pod "97ea71c9-692d-49fb-a079-183bda697421" (UID: "97ea71c9-692d-49fb-a079-183bda697421"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:53:52.073085 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:52.073062    4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97ea71c9-692d-49fb-a079-183bda697421-plugins-config-volume\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:53:52.075076 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:52.075051    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97ea71c9-692d-49fb-a079-183bda697421-kube-api-access-xkwgv" pod "97ea71c9-692d-49fb-a079-183bda697421" (UID: "97ea71c9-692d-49fb-a079-183bda697421"). InnerVolumeSpecName "kube-api-access-xkwgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:53:52.174041 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:52.174007    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16008341-844a-48a1-80dd-9f804fbcae4a-kube-api-access-cbt6m\" (UniqueName: \"kubernetes.io/projected/16008341-844a-48a1-80dd-9f804fbcae4a-kube-api-access-cbt6m\") pod \"16008341-844a-48a1-80dd-9f804fbcae4a\" (UID: \"16008341-844a-48a1-80dd-9f804fbcae4a\") "
Nov 03 22:53:52.174333 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:52.174305    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16008341-844a-48a1-80dd-9f804fbcae4a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16008341-844a-48a1-80dd-9f804fbcae4a-plugins-config-volume\") pod \"16008341-844a-48a1-80dd-9f804fbcae4a\" (UID: \"16008341-844a-48a1-80dd-9f804fbcae4a\") "
Nov 03 22:53:52.174768 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:52.174739    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16008341-844a-48a1-80dd-9f804fbcae4a-plugins-config-volume" pod "16008341-844a-48a1-80dd-9f804fbcae4a" (UID: "16008341-844a-48a1-80dd-9f804fbcae4a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:53:52.174980 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:52.174964    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkwgv\" (UniqueName: \"kubernetes.io/projected/97ea71c9-692d-49fb-a079-183bda697421-kube-api-access-xkwgv\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:53:52.175069 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:52.174982    4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16008341-844a-48a1-80dd-9f804fbcae4a-plugins-config-volume\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:53:52.176342 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:52.176319    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16008341-844a-48a1-80dd-9f804fbcae4a-kube-api-access-cbt6m" pod "16008341-844a-48a1-80dd-9f804fbcae4a" (UID: "16008341-844a-48a1-80dd-9f804fbcae4a"). InnerVolumeSpecName "kube-api-access-cbt6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:53:52.276856 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:52.276763    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbt6m\" (UniqueName: \"kubernetes.io/projected/16008341-844a-48a1-80dd-9f804fbcae4a-kube-api-access-cbt6m\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:53:52.371029 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:52.370989    4431 generic.go:356] "Generic (PLEG): container finished" podID="97ea71c9-692d-49fb-a079-183bda697421" containerID="f285c04e5ad7f1b0e10d0bf65034d1717b76113e91a72f5b07a7fa8804242eec" exitCode=0
Nov 03 22:53:52.371256 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:52.371040    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-341c51-857b566498-5d2tg"
Nov 03 22:53:52.371256 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:52.371080    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-341c51-857b566498-5d2tg" event={"ID":"97ea71c9-692d-49fb-a079-183bda697421","Type":"ContainerDied","Data":"f285c04e5ad7f1b0e10d0bf65034d1717b76113e91a72f5b07a7fa8804242eec"}
Nov 03 22:53:52.371256 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:52.371131    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-341c51-857b566498-5d2tg" event={"ID":"97ea71c9-692d-49fb-a079-183bda697421","Type":"ContainerDied","Data":"42b77a26829cd07e9133b01dd6258056cea9d7328cb4984695a1db45885eef8d"}
Nov 03 22:53:52.371256 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:52.371156    4431 scope.go:122] "RemoveContainer" containerID="f285c04e5ad7f1b0e10d0bf65034d1717b76113e91a72f5b07a7fa8804242eec"
Nov 03 22:53:52.378999 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:52.376022    4431 generic.go:356] "Generic (PLEG): container finished" podID="16008341-844a-48a1-80dd-9f804fbcae4a" containerID="409774e8ed984be083cbd48927211d23ba434226afca97ce4858a73fa9c8003e" exitCode=0
Nov 03 22:53:52.378999 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:52.376285    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-341c51-76d98b8f6c-cqm9p" event={"ID":"16008341-844a-48a1-80dd-9f804fbcae4a","Type":"ContainerDied","Data":"409774e8ed984be083cbd48927211d23ba434226afca97ce4858a73fa9c8003e"}
Nov 03 22:53:52.378999 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:52.376362    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-341c51-76d98b8f6c-cqm9p" event={"ID":"16008341-844a-48a1-80dd-9f804fbcae4a","Type":"ContainerDied","Data":"f37d2f5c7798eb28ff2ab2746a943e66a2784f8a8332d381791d092718e88401"}
Nov 03 22:53:52.378999 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:52.376473    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-341c51-76d98b8f6c-cqm9p"
Nov 03 22:53:52.394726 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:52.394702    4431 scope.go:122] "RemoveContainer" containerID="f285c04e5ad7f1b0e10d0bf65034d1717b76113e91a72f5b07a7fa8804242eec"
Nov 03 22:53:52.395025 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:53:52.395002    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f285c04e5ad7f1b0e10d0bf65034d1717b76113e91a72f5b07a7fa8804242eec\": container with ID starting with f285c04e5ad7f1b0e10d0bf65034d1717b76113e91a72f5b07a7fa8804242eec not found: ID does not exist" containerID="f285c04e5ad7f1b0e10d0bf65034d1717b76113e91a72f5b07a7fa8804242eec"
Nov 03 22:53:52.395102 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:52.395034    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f285c04e5ad7f1b0e10d0bf65034d1717b76113e91a72f5b07a7fa8804242eec"} err="failed to get container status \"f285c04e5ad7f1b0e10d0bf65034d1717b76113e91a72f5b07a7fa8804242eec\": rpc error: code = NotFound desc = could not find container \"f285c04e5ad7f1b0e10d0bf65034d1717b76113e91a72f5b07a7fa8804242eec\": container with ID starting with f285c04e5ad7f1b0e10d0bf65034d1717b76113e91a72f5b07a7fa8804242eec not found: ID does not exist"
Nov 03 22:53:52.395102 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:52.395052    4431 scope.go:122] "RemoveContainer" containerID="409774e8ed984be083cbd48927211d23ba434226afca97ce4858a73fa9c8003e"
Nov 03 22:53:52.404951 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:52.404933    4431 scope.go:122] "RemoveContainer" containerID="409774e8ed984be083cbd48927211d23ba434226afca97ce4858a73fa9c8003e"
Nov 03 22:53:52.405246 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:53:52.405204    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"409774e8ed984be083cbd48927211d23ba434226afca97ce4858a73fa9c8003e\": container with ID starting with 409774e8ed984be083cbd48927211d23ba434226afca97ce4858a73fa9c8003e not found: ID does not exist" containerID="409774e8ed984be083cbd48927211d23ba434226afca97ce4858a73fa9c8003e"
Nov 03 22:53:52.405292 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:52.405260    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"409774e8ed984be083cbd48927211d23ba434226afca97ce4858a73fa9c8003e"} err="failed to get container status \"409774e8ed984be083cbd48927211d23ba434226afca97ce4858a73fa9c8003e\": rpc error: code = NotFound desc = could not find container \"409774e8ed984be083cbd48927211d23ba434226afca97ce4858a73fa9c8003e\": container with ID starting with 409774e8ed984be083cbd48927211d23ba434226afca97ce4858a73fa9c8003e not found: ID does not exist"
Nov 03 22:53:52.442114 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:52.442084    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-341c51-857b566498-5d2tg"]
Nov 03 22:53:52.445600 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:52.445578    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-341c51-857b566498-5d2tg"]
Nov 03 22:53:52.479418 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:52.479387    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-341c51-76d98b8f6c-cqm9p"]
Nov 03 22:53:52.488608 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:52.488577    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-341c51-76d98b8f6c-cqm9p"]
Nov 03 22:53:53.419417 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:53.419371    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:53:54.090368 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:54.090337    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16008341-844a-48a1-80dd-9f804fbcae4a" path="/var/lib/kubelet/pods/16008341-844a-48a1-80dd-9f804fbcae4a/volumes"
Nov 03 22:53:54.090821 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:54.090807    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97ea71c9-692d-49fb-a079-183bda697421" path="/var/lib/kubelet/pods/97ea71c9-692d-49fb-a079-183bda697421/volumes"
Nov 03 22:53:54.301543 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:54.301495    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:53:55.123422 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:55.123382    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:53:56.609598 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:56.609555    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:53:57.052557 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:57.052521    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:53:58.196447 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:58.196412    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:53:59.756710 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:53:59.756673    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:54:00.942061 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:00.942019    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-d8a3d1/auth-test-model-d8a3d1-kserve-5fbc7778b9-kj9kw"]
Nov 03 22:54:00.943119 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:00.943087    4431 cpu_manager.go:414] "containerMap: removing container" podUID="16008341-844a-48a1-80dd-9f804fbcae4a" containerName="payload-pre-processing"
Nov 03 22:54:00.943119 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:00.943118    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16008341-844a-48a1-80dd-9f804fbcae4a" containerName="payload-pre-processing"
Nov 03 22:54:00.943323 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:00.943171    4431 cpu_manager.go:414] "containerMap: removing container" podUID="97ea71c9-692d-49fb-a079-183bda697421" containerName="payload-processing"
Nov 03 22:54:00.943323 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:00.943179    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97ea71c9-692d-49fb-a079-183bda697421" containerName="payload-processing"
Nov 03 22:54:00.943429 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:00.943383    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="16008341-844a-48a1-80dd-9f804fbcae4a" containerName="payload-pre-processing"
Nov 03 22:54:00.943429 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:00.943405    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="97ea71c9-692d-49fb-a079-183bda697421" containerName="payload-processing"
Nov 03 22:54:00.948947 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:00.948926    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d8a3d1/auth-test-model-d8a3d1-kserve-5fbc7778b9-kj9kw"
Nov 03 22:54:00.963283 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:00.963248    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d8a3d1/auth-test-model-d8a3d1-kserve-5fbc7778b9-kj9kw"]
Nov 03 22:54:00.970037 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:00.970009    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-d8a3d1\"/\"openshift-service-ca.crt\""
Nov 03 22:54:00.970307 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:00.970288    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-d8a3d1\"/\"kube-root-ca.crt\""
Nov 03 22:54:00.978018 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:00.977996    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-d8a3d1\"/\"default-dockercfg-qbcsv\""
Nov 03 22:54:00.978144 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:00.978039    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-d8a3d1\"/\"auth-test-model-d8a3d1-kserve-self-signed-certs\""
Nov 03 22:54:01.034471 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:01.034440    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1503c14c-4dab-435d-b29f-5c60d0c0acfb-dshm\") pod \"auth-test-model-d8a3d1-kserve-5fbc7778b9-kj9kw\" (UID: \"1503c14c-4dab-435d-b29f-5c60d0c0acfb\") " pod="ai-tenant-e2e-shared-b-w3-d8a3d1/auth-test-model-d8a3d1-kserve-5fbc7778b9-kj9kw"
Nov 03 22:54:01.034647 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:01.034484    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95g99\" (UniqueName: \"kubernetes.io/projected/1503c14c-4dab-435d-b29f-5c60d0c0acfb-kube-api-access-95g99\") pod \"auth-test-model-d8a3d1-kserve-5fbc7778b9-kj9kw\" (UID: \"1503c14c-4dab-435d-b29f-5c60d0c0acfb\") " pod="ai-tenant-e2e-shared-b-w3-d8a3d1/auth-test-model-d8a3d1-kserve-5fbc7778b9-kj9kw"
Nov 03 22:54:01.034752 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:01.034686    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1503c14c-4dab-435d-b29f-5c60d0c0acfb-tmp-dir\") pod \"auth-test-model-d8a3d1-kserve-5fbc7778b9-kj9kw\" (UID: \"1503c14c-4dab-435d-b29f-5c60d0c0acfb\") " pod="ai-tenant-e2e-shared-b-w3-d8a3d1/auth-test-model-d8a3d1-kserve-5fbc7778b9-kj9kw"
Nov 03 22:54:01.034752 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:01.034729    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1503c14c-4dab-435d-b29f-5c60d0c0acfb-model-cache\") pod \"auth-test-model-d8a3d1-kserve-5fbc7778b9-kj9kw\" (UID: \"1503c14c-4dab-435d-b29f-5c60d0c0acfb\") " pod="ai-tenant-e2e-shared-b-w3-d8a3d1/auth-test-model-d8a3d1-kserve-5fbc7778b9-kj9kw"
Nov 03 22:54:01.034844 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:01.034832    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1503c14c-4dab-435d-b29f-5c60d0c0acfb-home\") pod \"auth-test-model-d8a3d1-kserve-5fbc7778b9-kj9kw\" (UID: \"1503c14c-4dab-435d-b29f-5c60d0c0acfb\") " pod="ai-tenant-e2e-shared-b-w3-d8a3d1/auth-test-model-d8a3d1-kserve-5fbc7778b9-kj9kw"
Nov 03 22:54:01.034933 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:01.034918    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1503c14c-4dab-435d-b29f-5c60d0c0acfb-tls-certs\") pod \"auth-test-model-d8a3d1-kserve-5fbc7778b9-kj9kw\" (UID: \"1503c14c-4dab-435d-b29f-5c60d0c0acfb\") " pod="ai-tenant-e2e-shared-b-w3-d8a3d1/auth-test-model-d8a3d1-kserve-5fbc7778b9-kj9kw"
Nov 03 22:54:01.039409 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:01.039384    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:54:01.136569 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:01.136526    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1503c14c-4dab-435d-b29f-5c60d0c0acfb-dshm\") pod \"auth-test-model-d8a3d1-kserve-5fbc7778b9-kj9kw\" (UID: \"1503c14c-4dab-435d-b29f-5c60d0c0acfb\") " pod="ai-tenant-e2e-shared-b-w3-d8a3d1/auth-test-model-d8a3d1-kserve-5fbc7778b9-kj9kw"
Nov 03 22:54:01.136761 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:01.136602    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95g99\" (UniqueName: \"kubernetes.io/projected/1503c14c-4dab-435d-b29f-5c60d0c0acfb-kube-api-access-95g99\") pod \"auth-test-model-d8a3d1-kserve-5fbc7778b9-kj9kw\" (UID: \"1503c14c-4dab-435d-b29f-5c60d0c0acfb\") " pod="ai-tenant-e2e-shared-b-w3-d8a3d1/auth-test-model-d8a3d1-kserve-5fbc7778b9-kj9kw"
Nov 03 22:54:01.136830 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:01.136761    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1503c14c-4dab-435d-b29f-5c60d0c0acfb-tmp-dir\") pod \"auth-test-model-d8a3d1-kserve-5fbc7778b9-kj9kw\" (UID: \"1503c14c-4dab-435d-b29f-5c60d0c0acfb\") " pod="ai-tenant-e2e-shared-b-w3-d8a3d1/auth-test-model-d8a3d1-kserve-5fbc7778b9-kj9kw"
Nov 03 22:54:01.136830 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:01.136793    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1503c14c-4dab-435d-b29f-5c60d0c0acfb-model-cache\") pod \"auth-test-model-d8a3d1-kserve-5fbc7778b9-kj9kw\" (UID: \"1503c14c-4dab-435d-b29f-5c60d0c0acfb\") " pod="ai-tenant-e2e-shared-b-w3-d8a3d1/auth-test-model-d8a3d1-kserve-5fbc7778b9-kj9kw"
Nov 03 22:54:01.136943 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:01.136861    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1503c14c-4dab-435d-b29f-5c60d0c0acfb-home\") pod \"auth-test-model-d8a3d1-kserve-5fbc7778b9-kj9kw\" (UID: \"1503c14c-4dab-435d-b29f-5c60d0c0acfb\") " pod="ai-tenant-e2e-shared-b-w3-d8a3d1/auth-test-model-d8a3d1-kserve-5fbc7778b9-kj9kw"
Nov 03 22:54:01.136943 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:01.136936    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1503c14c-4dab-435d-b29f-5c60d0c0acfb-tls-certs\") pod \"auth-test-model-d8a3d1-kserve-5fbc7778b9-kj9kw\" (UID: \"1503c14c-4dab-435d-b29f-5c60d0c0acfb\") " pod="ai-tenant-e2e-shared-b-w3-d8a3d1/auth-test-model-d8a3d1-kserve-5fbc7778b9-kj9kw"
Nov 03 22:54:01.137320 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:01.137296    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1503c14c-4dab-435d-b29f-5c60d0c0acfb-model-cache\") pod \"auth-test-model-d8a3d1-kserve-5fbc7778b9-kj9kw\" (UID: \"1503c14c-4dab-435d-b29f-5c60d0c0acfb\") " pod="ai-tenant-e2e-shared-b-w3-d8a3d1/auth-test-model-d8a3d1-kserve-5fbc7778b9-kj9kw"
Nov 03 22:54:01.137543 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:01.137519    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1503c14c-4dab-435d-b29f-5c60d0c0acfb-tmp-dir\") pod \"auth-test-model-d8a3d1-kserve-5fbc7778b9-kj9kw\" (UID: \"1503c14c-4dab-435d-b29f-5c60d0c0acfb\") " pod="ai-tenant-e2e-shared-b-w3-d8a3d1/auth-test-model-d8a3d1-kserve-5fbc7778b9-kj9kw"
Nov 03 22:54:01.137836 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:01.137760    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1503c14c-4dab-435d-b29f-5c60d0c0acfb-home\") pod \"auth-test-model-d8a3d1-kserve-5fbc7778b9-kj9kw\" (UID: \"1503c14c-4dab-435d-b29f-5c60d0c0acfb\") " pod="ai-tenant-e2e-shared-b-w3-d8a3d1/auth-test-model-d8a3d1-kserve-5fbc7778b9-kj9kw"
Nov 03 22:54:01.139394 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:01.139367    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1503c14c-4dab-435d-b29f-5c60d0c0acfb-dshm\") pod \"auth-test-model-d8a3d1-kserve-5fbc7778b9-kj9kw\" (UID: \"1503c14c-4dab-435d-b29f-5c60d0c0acfb\") " pod="ai-tenant-e2e-shared-b-w3-d8a3d1/auth-test-model-d8a3d1-kserve-5fbc7778b9-kj9kw"
Nov 03 22:54:01.140786 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:01.140764    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1503c14c-4dab-435d-b29f-5c60d0c0acfb-tls-certs\") pod \"auth-test-model-d8a3d1-kserve-5fbc7778b9-kj9kw\" (UID: \"1503c14c-4dab-435d-b29f-5c60d0c0acfb\") " pod="ai-tenant-e2e-shared-b-w3-d8a3d1/auth-test-model-d8a3d1-kserve-5fbc7778b9-kj9kw"
Nov 03 22:54:01.159577 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:01.159552    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95g99\" (UniqueName: \"kubernetes.io/projected/1503c14c-4dab-435d-b29f-5c60d0c0acfb-kube-api-access-95g99\") pod \"auth-test-model-d8a3d1-kserve-5fbc7778b9-kj9kw\" (UID: \"1503c14c-4dab-435d-b29f-5c60d0c0acfb\") " pod="ai-tenant-e2e-shared-b-w3-d8a3d1/auth-test-model-d8a3d1-kserve-5fbc7778b9-kj9kw"
Nov 03 22:54:01.267070 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:01.266971    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d8a3d1/auth-test-model-d8a3d1-kserve-5fbc7778b9-kj9kw"
Nov 03 22:54:01.289300 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:54:01.289264    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:54:01.422919 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:01.422882    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:54:01.583500 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:01.583470    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d8a3d1/auth-test-model-d8a3d1-kserve-5fbc7778b9-kj9kw"]
Nov 03 22:54:01.585488 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:54:01.585450    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1503c14c_4dab_435d_b29f_5c60d0c0acfb.slice/crio-4b407ea464c5f36b7c82c53ab6f7b342f029c959769a2fb6dfda985caeae3d9c WatchSource:0}: Error finding container 4b407ea464c5f36b7c82c53ab6f7b342f029c959769a2fb6dfda985caeae3d9c: Status 404 returned error can't find the container with id 4b407ea464c5f36b7c82c53ab6f7b342f029c959769a2fb6dfda985caeae3d9c
Nov 03 22:54:02.466845 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:02.466803    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d8a3d1/auth-test-model-d8a3d1-kserve-5fbc7778b9-kj9kw" event={"ID":"1503c14c-4dab-435d-b29f-5c60d0c0acfb","Type":"ContainerStarted","Data":"b28828735935672e686447cafd01882daefadfbcdba667ebeff32f5b0f4f94fd"}
Nov 03 22:54:02.467275 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:02.466856    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d8a3d1/auth-test-model-d8a3d1-kserve-5fbc7778b9-kj9kw" event={"ID":"1503c14c-4dab-435d-b29f-5c60d0c0acfb","Type":"ContainerStarted","Data":"4b407ea464c5f36b7c82c53ab6f7b342f029c959769a2fb6dfda985caeae3d9c"}
Nov 03 22:54:02.509384 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:02.509325    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-d8a3d1/auth-test-model-d8a3d1-kserve-5fbc7778b9-kj9kw" podStartSLOduration=2.509310504 podStartE2EDuration="2.509310504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:54:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:54:02.507040517 +0000 UTC m=+2466.532896381" watchObservedRunningTime="2026-11-03 22:54:02.509310504 +0000 UTC m=+2466.535166424"
Nov 03 22:54:03.011816 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:03.011783    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:54:03.268117 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:03.268040    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-d8a3d1/auth-test-model-d8a3d1-kserve-5fbc7778b9-kj9kw"
Nov 03 22:54:04.383164 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:04.383115    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:54:04.946376 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:04.946333    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:54:05.088388 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:05.088355    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:54:05.520123 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:05.520078    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:54:06.792551 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:06.792513    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:54:08.380335 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:08.380295    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:54:09.916707 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:09.916672    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:54:11.267596 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:11.267555    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-d8a3d1/auth-test-model-d8a3d1-kserve-5fbc7778b9-kj9kw"
Nov 03 22:54:11.280532 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:11.280509    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-d8a3d1/auth-test-model-d8a3d1-kserve-5fbc7778b9-kj9kw"
Nov 03 22:54:11.541494 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:11.541390    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:54:11.560508 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:11.560475    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-d8a3d1/auth-test-model-d8a3d1-kserve-5fbc7778b9-kj9kw"
Nov 03 22:54:12.293854 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:12.293812    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:54:12.440425 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:12.440392    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:54:12.633181 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:12.633089    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:54:13.132263 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:13.132228    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:54:14.904364 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:14.904332    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:54:16.503121 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:16.503085    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:54:18.121189 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:18.121150    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:54:19.696555 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:19.696502    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:54:19.846710 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:19.846674    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:54:21.412263 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:21.412198    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:54:23.050055 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:23.050021    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:54:23.603170 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:23.603137    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-124490-7cbff46899-7c5p6"]
Nov 03 22:54:23.737570 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:23.737498    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:54:23.893198 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:23.893113    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:54:24.039496 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:24.039460    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:54:24.634721 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:24.634683    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:54:26.230032 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:26.229960    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:54:27.824770 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:27.824736    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:54:28.603779 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:28.603738    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-124490-7cbff46899-7c5p6" podUID="fffa2baf-eaca-4459-b8c4-63ded79ac8ec" containerName="payload-pre-processing" containerID="cri-o://f1bf17e8fb23a68a85b76df4d96f59f886a0b5f682a5efe587ab02b34eaba549" gracePeriod=25
Nov 03 22:54:29.310225 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:29.310181    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:54:30.552572 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:30.552527    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:54:30.694626 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:30.694592    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:54:30.909611 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:30.909524    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:54:31.260937 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:54:31.260901    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:54:31.368669 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:31.368608    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:54:32.264101 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:32.264055    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:54:32.512479 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:32.512447    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:54:33.509169 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:33.509133    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-124490-7cbff46899-7c5p6" podUID="fffa2baf-eaca-4459-b8c4-63ded79ac8ec" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 22:54:34.019333 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:34.019299    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:54:35.351002 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:35.350965    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:54:35.678878 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:35.678842    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:54:37.077903 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:37.077867    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:54:37.620108 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:37.620076    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:54:37.780011 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:37.779970    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:54:38.187722 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:38.187673    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:54:39.421465 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:39.421428    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:54:41.112908 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:41.112859    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:54:42.804989 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:42.804950    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:54:43.235891 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:43.235852    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:54:43.508891 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:43.508793    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-124490-7cbff46899-7c5p6" podUID="fffa2baf-eaca-4459-b8c4-63ded79ac8ec" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 22:54:44.407438 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:44.407399    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:54:44.550530 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:44.550499    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:54:44.695799 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:44.695767    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:54:45.246653 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:45.246618    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:54:45.427781 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:45.427727    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:54:45.646525 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:45.646442    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq"]
Nov 03 22:54:45.646780 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:45.646748    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq" podUID="51131709-120f-4b00-a6ec-34e879ac168b" containerName="istio-proxy" containerID="cri-o://254a92a8fc5eb9d34807583d4d95b4862e8f0d42259922f29746d533d66b4d49" gracePeriod=30
Nov 03 22:54:45.736927 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:45.736887    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:54:46.573552 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:46.573517    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:54:48.210725 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:48.210678    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:54:48.861953 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:48.861882    4431 generic.go:356] "Generic (PLEG): container finished" podID="fffa2baf-eaca-4459-b8c4-63ded79ac8ec" containerID="f1bf17e8fb23a68a85b76df4d96f59f886a0b5f682a5efe587ab02b34eaba549" exitCode=0
Nov 03 22:54:48.862068 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:48.861960    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-124490-7cbff46899-7c5p6" event={"ID":"fffa2baf-eaca-4459-b8c4-63ded79ac8ec","Type":"ContainerDied","Data":"f1bf17e8fb23a68a85b76df4d96f59f886a0b5f682a5efe587ab02b34eaba549"}
Nov 03 22:54:48.894812 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:48.894780    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-124490-7cbff46899-7c5p6"
Nov 03 22:54:48.996377 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:48.996345    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fffa2baf-eaca-4459-b8c4-63ded79ac8ec-kube-api-access-m7vww\" (UniqueName: \"kubernetes.io/projected/fffa2baf-eaca-4459-b8c4-63ded79ac8ec-kube-api-access-m7vww\") pod \"fffa2baf-eaca-4459-b8c4-63ded79ac8ec\" (UID: \"fffa2baf-eaca-4459-b8c4-63ded79ac8ec\") "
Nov 03 22:54:48.996557 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:48.996519    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fffa2baf-eaca-4459-b8c4-63ded79ac8ec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fffa2baf-eaca-4459-b8c4-63ded79ac8ec-plugins-config-volume\") pod \"fffa2baf-eaca-4459-b8c4-63ded79ac8ec\" (UID: \"fffa2baf-eaca-4459-b8c4-63ded79ac8ec\") "
Nov 03 22:54:48.996884 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:48.996859    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fffa2baf-eaca-4459-b8c4-63ded79ac8ec-plugins-config-volume" pod "fffa2baf-eaca-4459-b8c4-63ded79ac8ec" (UID: "fffa2baf-eaca-4459-b8c4-63ded79ac8ec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:54:48.998621 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:48.998600    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fffa2baf-eaca-4459-b8c4-63ded79ac8ec-kube-api-access-m7vww" pod "fffa2baf-eaca-4459-b8c4-63ded79ac8ec" (UID: "fffa2baf-eaca-4459-b8c4-63ded79ac8ec"). InnerVolumeSpecName "kube-api-access-m7vww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:54:49.097789 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:49.097753    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7vww\" (UniqueName: \"kubernetes.io/projected/fffa2baf-eaca-4459-b8c4-63ded79ac8ec-kube-api-access-m7vww\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:54:49.097789 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:49.097782    4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fffa2baf-eaca-4459-b8c4-63ded79ac8ec-plugins-config-volume\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:54:49.266857 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:49.266815    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:54:49.486556 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:49.486513    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:54:49.912715 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:49.912673    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-124490-7cbff46899-7c5p6" event={"ID":"fffa2baf-eaca-4459-b8c4-63ded79ac8ec","Type":"ContainerDied","Data":"270f9230be844fa7a769d412bfc477b36164cbab679a61770be3df12de19b26e"}
Nov 03 22:54:49.912925 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:49.912793    4431 scope.go:122] "RemoveContainer" containerID="f1bf17e8fb23a68a85b76df4d96f59f886a0b5f682a5efe587ab02b34eaba549"
Nov 03 22:54:49.913009 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:49.912979    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-124490-7cbff46899-7c5p6"
Nov 03 22:54:49.970278 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:49.970244    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-124490-7cbff46899-7c5p6"]
Nov 03 22:54:49.980172 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:49.980138    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:54:49.983352 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:49.983323    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-124490-7cbff46899-7c5p6"]
Nov 03 22:54:50.089798 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:50.089763    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fffa2baf-eaca-4459-b8c4-63ded79ac8ec" path="/var/lib/kubelet/pods/fffa2baf-eaca-4459-b8c4-63ded79ac8ec/volumes"
Nov 03 22:54:50.123277 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:50.123190    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:54:50.565043 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:50.564986    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:54:50.923436 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:50.923408    4431 generic.go:356] "Generic (PLEG): container finished" podID="51131709-120f-4b00-a6ec-34e879ac168b" containerID="254a92a8fc5eb9d34807583d4d95b4862e8f0d42259922f29746d533d66b4d49" exitCode=0
Nov 03 22:54:50.923592 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:50.923450    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq" event={"ID":"51131709-120f-4b00-a6ec-34e879ac168b","Type":"ContainerDied","Data":"254a92a8fc5eb9d34807583d4d95b4862e8f0d42259922f29746d533d66b4d49"}
Nov 03 22:54:50.923592 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:50.923504    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq" event={"ID":"51131709-120f-4b00-a6ec-34e879ac168b","Type":"ContainerDied","Data":"d550ba28fb462316335ae684aaf23bdb475a557d6b2a460633eecb70029369bb"}
Nov 03 22:54:50.923592 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:50.923521    4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d550ba28fb462316335ae684aaf23bdb475a557d6b2a460633eecb70029369bb"
Nov 03 22:54:50.923592 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:50.923534    4431 scope.go:122] "RemoveContainer" containerID="33077418fd8a5ad61a910ab6c88db15081db2621f5b0591440c405b460d30f02"
Nov 03 22:54:50.925840 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:50.925823    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq"
Nov 03 22:54:51.037700 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:51.037668    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51131709-120f-4b00-a6ec-34e879ac168b-istio-token\" (UniqueName: \"kubernetes.io/projected/51131709-120f-4b00-a6ec-34e879ac168b-istio-token\") pod \"51131709-120f-4b00-a6ec-34e879ac168b\" (UID: \"51131709-120f-4b00-a6ec-34e879ac168b\") "
Nov 03 22:54:51.037886 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:51.037712    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51131709-120f-4b00-a6ec-34e879ac168b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51131709-120f-4b00-a6ec-34e879ac168b-istiod-ca-cert\") pod \"51131709-120f-4b00-a6ec-34e879ac168b\" (UID: \"51131709-120f-4b00-a6ec-34e879ac168b\") "
Nov 03 22:54:51.037886 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:51.037804    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51131709-120f-4b00-a6ec-34e879ac168b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51131709-120f-4b00-a6ec-34e879ac168b-workload-certs\") pod \"51131709-120f-4b00-a6ec-34e879ac168b\" (UID: \"51131709-120f-4b00-a6ec-34e879ac168b\") "
Nov 03 22:54:51.037973 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:51.037894    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51131709-120f-4b00-a6ec-34e879ac168b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51131709-120f-4b00-a6ec-34e879ac168b-istio-envoy\") pod \"51131709-120f-4b00-a6ec-34e879ac168b\" (UID: \"51131709-120f-4b00-a6ec-34e879ac168b\") "
Nov 03 22:54:51.038015 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:51.037985    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51131709-120f-4b00-a6ec-34e879ac168b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51131709-120f-4b00-a6ec-34e879ac168b-credential-socket\") pod \"51131709-120f-4b00-a6ec-34e879ac168b\" (UID: \"51131709-120f-4b00-a6ec-34e879ac168b\") "
Nov 03 22:54:51.038099 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:51.038083    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/51131709-120f-4b00-a6ec-34e879ac168b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51131709-120f-4b00-a6ec-34e879ac168b-istio-podinfo\") pod \"51131709-120f-4b00-a6ec-34e879ac168b\" (UID: \"51131709-120f-4b00-a6ec-34e879ac168b\") "
Nov 03 22:54:51.038228 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:51.038201    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51131709-120f-4b00-a6ec-34e879ac168b-kube-api-access-pmt69\" (UniqueName: \"kubernetes.io/projected/51131709-120f-4b00-a6ec-34e879ac168b-kube-api-access-pmt69\") pod \"51131709-120f-4b00-a6ec-34e879ac168b\" (UID: \"51131709-120f-4b00-a6ec-34e879ac168b\") "
Nov 03 22:54:51.038305 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:51.038086    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51131709-120f-4b00-a6ec-34e879ac168b-workload-certs" pod "51131709-120f-4b00-a6ec-34e879ac168b" (UID: "51131709-120f-4b00-a6ec-34e879ac168b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:54:51.038360 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:51.038293    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51131709-120f-4b00-a6ec-34e879ac168b-istiod-ca-cert" pod "51131709-120f-4b00-a6ec-34e879ac168b" (UID: "51131709-120f-4b00-a6ec-34e879ac168b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:54:51.038360 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:51.038275    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51131709-120f-4b00-a6ec-34e879ac168b-credential-socket" pod "51131709-120f-4b00-a6ec-34e879ac168b" (UID: "51131709-120f-4b00-a6ec-34e879ac168b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:54:51.038470 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:51.038386    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51131709-120f-4b00-a6ec-34e879ac168b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51131709-120f-4b00-a6ec-34e879ac168b-istio-data\") pod \"51131709-120f-4b00-a6ec-34e879ac168b\" (UID: \"51131709-120f-4b00-a6ec-34e879ac168b\") "
Nov 03 22:54:51.038470 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:51.038419    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51131709-120f-4b00-a6ec-34e879ac168b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51131709-120f-4b00-a6ec-34e879ac168b-workload-socket\") pod \"51131709-120f-4b00-a6ec-34e879ac168b\" (UID: \"51131709-120f-4b00-a6ec-34e879ac168b\") "
Nov 03 22:54:51.038750 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:51.038726    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51131709-120f-4b00-a6ec-34e879ac168b-workload-socket" pod "51131709-120f-4b00-a6ec-34e879ac168b" (UID: "51131709-120f-4b00-a6ec-34e879ac168b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:54:51.039255 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:51.039140    4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51131709-120f-4b00-a6ec-34e879ac168b-workload-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:54:51.039255 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:51.039168    4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51131709-120f-4b00-a6ec-34e879ac168b-istiod-ca-cert\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:54:51.039255 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:51.039183    4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51131709-120f-4b00-a6ec-34e879ac168b-workload-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:54:51.039255 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:51.039196    4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51131709-120f-4b00-a6ec-34e879ac168b-credential-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:54:51.039782 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:51.039746    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51131709-120f-4b00-a6ec-34e879ac168b-istio-data" pod "51131709-120f-4b00-a6ec-34e879ac168b" (UID: "51131709-120f-4b00-a6ec-34e879ac168b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:54:51.040248 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:51.040198    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51131709-120f-4b00-a6ec-34e879ac168b-istio-token" pod "51131709-120f-4b00-a6ec-34e879ac168b" (UID: "51131709-120f-4b00-a6ec-34e879ac168b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:54:51.040434 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:51.040418    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51131709-120f-4b00-a6ec-34e879ac168b-kube-api-access-pmt69" pod "51131709-120f-4b00-a6ec-34e879ac168b" (UID: "51131709-120f-4b00-a6ec-34e879ac168b"). InnerVolumeSpecName "kube-api-access-pmt69". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:54:51.040503 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:51.040460    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/51131709-120f-4b00-a6ec-34e879ac168b-istio-podinfo" pod "51131709-120f-4b00-a6ec-34e879ac168b" (UID: "51131709-120f-4b00-a6ec-34e879ac168b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 22:54:51.040623 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:51.040602    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51131709-120f-4b00-a6ec-34e879ac168b-istio-envoy" pod "51131709-120f-4b00-a6ec-34e879ac168b" (UID: "51131709-120f-4b00-a6ec-34e879ac168b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:54:51.140740 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:51.140710    4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51131709-120f-4b00-a6ec-34e879ac168b-istio-podinfo\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:54:51.140740 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:51.140737    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmt69\" (UniqueName: \"kubernetes.io/projected/51131709-120f-4b00-a6ec-34e879ac168b-kube-api-access-pmt69\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:54:51.140740 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:51.140747    4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51131709-120f-4b00-a6ec-34e879ac168b-istio-data\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:54:51.140976 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:51.140757    4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51131709-120f-4b00-a6ec-34e879ac168b-istio-token\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:54:51.140976 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:51.140764    4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51131709-120f-4b00-a6ec-34e879ac168b-istio-envoy\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:54:51.800664 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:51.800626    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:54:51.931976 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:51.931942    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq"
Nov 03 22:54:52.034173 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:52.034142    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq"]
Nov 03 22:54:52.038532 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:52.038507    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-bb1fa5-openshift-default-95cc6c6f-8b6vq"]
Nov 03 22:54:52.088601 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:52.088518    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51131709-120f-4b00-a6ec-34e879ac168b" path="/var/lib/kubelet/pods/51131709-120f-4b00-a6ec-34e879ac168b/volumes"
Nov 03 22:54:53.406681 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:53.406645    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:54:54.402579 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:54.402545    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:54:55.001971 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:55.001940    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:54:56.575566 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:56.575532    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:54:57.325117 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:57.325078    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:54:57.471655 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:57.471621    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:54:57.638422 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:57.638341    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:54:58.243696 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:58.243633    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:54:58.392163 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:58.392123    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:54:58.809603 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:58.809565    4431 scope.go:122] "RemoveContainer" containerID="254a92a8fc5eb9d34807583d4d95b4862e8f0d42259922f29746d533d66b4d49"
Nov 03 22:54:59.991689 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:54:59.991647    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:55:01.259968 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:55:01.259931    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:55:01.627136 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:01.627058    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:55:02.602585 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:02.602553    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:55:03.521265 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:03.521233    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:55:05.347821 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:05.347786    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:55:07.192406 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:07.192362    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:55:08.732369 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:08.732319    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:55:10.231864 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:10.231802    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:55:11.838021 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:11.837977    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:55:13.412266 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:13.412207    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:55:14.961853 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:14.961809    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:55:16.243204 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:16.243167    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:55:16.785398 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:16.785356    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:55:16.944950 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:16.944910    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:55:17.406187 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:17.406154    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:55:18.644361 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:18.644324    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:55:19.153228 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:19.153189    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5"]
Nov 03 22:55:19.153470 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:19.153443    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5" podUID="b37263ad-40ef-4f70-a5fd-a2a1f14a3534" containerName="istio-proxy" containerID="cri-o://6fd3924b880e294f5fcf394f52a0a8a1cdfa65e442e5b58c253c58ca7471b272" gracePeriod=30
Nov 03 22:55:20.113424 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:20.113388    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:55:21.616699 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:21.616533    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:55:22.755485 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:22.755448    4431 patch_prober.go:29] interesting pod/e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 22:55:22.755866 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:22.755507    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5" podUID="b37263ad-40ef-4f70-a5fd-a2a1f14a3534" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 22:55:23.050923 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:23.050824    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:55:23.859711 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:23.859670    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:55:24.064513 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:24.064445    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:55:24.210490 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:24.210459    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-963936-openshift-default-68b65f857c-phdvv"]
Nov 03 22:55:24.210736 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:24.210711    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-963936-openshift-default-68b65f857c-phdvv" podUID="2cbef074-f335-41d5-a69c-b84a958e87ee" containerName="istio-proxy" containerID="cri-o://ca8359caacf7375447f51fd89147358bd7a1ecd1353d57c6ef1171690d006cf7" gracePeriod=30
Nov 03 22:55:24.316093 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:24.316055    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:55:24.484828 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:24.484801    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5"
Nov 03 22:55:24.553734 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:24.553695    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b37263ad-40ef-4f70-a5fd-a2a1f14a3534-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b37263ad-40ef-4f70-a5fd-a2a1f14a3534-workload-socket\") pod \"b37263ad-40ef-4f70-a5fd-a2a1f14a3534\" (UID: \"b37263ad-40ef-4f70-a5fd-a2a1f14a3534\") "
Nov 03 22:55:24.553925 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:24.553749    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b37263ad-40ef-4f70-a5fd-a2a1f14a3534-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b37263ad-40ef-4f70-a5fd-a2a1f14a3534-istio-envoy\") pod \"b37263ad-40ef-4f70-a5fd-a2a1f14a3534\" (UID: \"b37263ad-40ef-4f70-a5fd-a2a1f14a3534\") "
Nov 03 22:55:24.553925 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:24.553791    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b37263ad-40ef-4f70-a5fd-a2a1f14a3534-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b37263ad-40ef-4f70-a5fd-a2a1f14a3534-credential-socket\") pod \"b37263ad-40ef-4f70-a5fd-a2a1f14a3534\" (UID: \"b37263ad-40ef-4f70-a5fd-a2a1f14a3534\") "
Nov 03 22:55:24.553925 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:24.553872    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b37263ad-40ef-4f70-a5fd-a2a1f14a3534-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b37263ad-40ef-4f70-a5fd-a2a1f14a3534-istio-data\") pod \"b37263ad-40ef-4f70-a5fd-a2a1f14a3534\" (UID: \"b37263ad-40ef-4f70-a5fd-a2a1f14a3534\") "
Nov 03 22:55:24.554124 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:24.553938    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b37263ad-40ef-4f70-a5fd-a2a1f14a3534-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b37263ad-40ef-4f70-a5fd-a2a1f14a3534-istio-podinfo\") pod \"b37263ad-40ef-4f70-a5fd-a2a1f14a3534\" (UID: \"b37263ad-40ef-4f70-a5fd-a2a1f14a3534\") "
Nov 03 22:55:24.554124 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:24.553995    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b37263ad-40ef-4f70-a5fd-a2a1f14a3534-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b37263ad-40ef-4f70-a5fd-a2a1f14a3534-workload-certs\") pod \"b37263ad-40ef-4f70-a5fd-a2a1f14a3534\" (UID: \"b37263ad-40ef-4f70-a5fd-a2a1f14a3534\") "
Nov 03 22:55:24.554253 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:24.554126    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b37263ad-40ef-4f70-a5fd-a2a1f14a3534-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b37263ad-40ef-4f70-a5fd-a2a1f14a3534-istiod-ca-cert\") pod \"b37263ad-40ef-4f70-a5fd-a2a1f14a3534\" (UID: \"b37263ad-40ef-4f70-a5fd-a2a1f14a3534\") "
Nov 03 22:55:24.554328 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:24.554259    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b37263ad-40ef-4f70-a5fd-a2a1f14a3534-credential-socket" pod "b37263ad-40ef-4f70-a5fd-a2a1f14a3534" (UID: "b37263ad-40ef-4f70-a5fd-a2a1f14a3534"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:55:24.554328 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:24.554310    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b37263ad-40ef-4f70-a5fd-a2a1f14a3534-kube-api-access-h969q\" (UniqueName: \"kubernetes.io/projected/b37263ad-40ef-4f70-a5fd-a2a1f14a3534-kube-api-access-h969q\") pod \"b37263ad-40ef-4f70-a5fd-a2a1f14a3534\" (UID: \"b37263ad-40ef-4f70-a5fd-a2a1f14a3534\") "
Nov 03 22:55:24.554449 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:24.554365    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b37263ad-40ef-4f70-a5fd-a2a1f14a3534-istio-token\" (UniqueName: \"kubernetes.io/projected/b37263ad-40ef-4f70-a5fd-a2a1f14a3534-istio-token\") pod \"b37263ad-40ef-4f70-a5fd-a2a1f14a3534\" (UID: \"b37263ad-40ef-4f70-a5fd-a2a1f14a3534\") "
Nov 03 22:55:24.555119 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:24.554528    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b37263ad-40ef-4f70-a5fd-a2a1f14a3534-workload-certs" pod "b37263ad-40ef-4f70-a5fd-a2a1f14a3534" (UID: "b37263ad-40ef-4f70-a5fd-a2a1f14a3534"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:55:24.555119 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:24.554587    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b37263ad-40ef-4f70-a5fd-a2a1f14a3534-workload-socket" pod "b37263ad-40ef-4f70-a5fd-a2a1f14a3534" (UID: "b37263ad-40ef-4f70-a5fd-a2a1f14a3534"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:55:24.555119 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:24.554630    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b37263ad-40ef-4f70-a5fd-a2a1f14a3534-istio-data" pod "b37263ad-40ef-4f70-a5fd-a2a1f14a3534" (UID: "b37263ad-40ef-4f70-a5fd-a2a1f14a3534"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:55:24.555119 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:24.554850    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b37263ad-40ef-4f70-a5fd-a2a1f14a3534-istiod-ca-cert" pod "b37263ad-40ef-4f70-a5fd-a2a1f14a3534" (UID: "b37263ad-40ef-4f70-a5fd-a2a1f14a3534"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:55:24.555119 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:24.555044    4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b37263ad-40ef-4f70-a5fd-a2a1f14a3534-istiod-ca-cert\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:55:24.555119 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:24.555060    4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b37263ad-40ef-4f70-a5fd-a2a1f14a3534-workload-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:55:24.555119 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:24.555072    4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b37263ad-40ef-4f70-a5fd-a2a1f14a3534-credential-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:55:24.555119 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:24.555085    4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b37263ad-40ef-4f70-a5fd-a2a1f14a3534-istio-data\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:55:24.555119 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:24.555097    4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b37263ad-40ef-4f70-a5fd-a2a1f14a3534-workload-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:55:24.556178 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:24.556153    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b37263ad-40ef-4f70-a5fd-a2a1f14a3534-istio-envoy" pod "b37263ad-40ef-4f70-a5fd-a2a1f14a3534" (UID: "b37263ad-40ef-4f70-a5fd-a2a1f14a3534"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:55:24.556739 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:24.556713    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b37263ad-40ef-4f70-a5fd-a2a1f14a3534-istio-podinfo" pod "b37263ad-40ef-4f70-a5fd-a2a1f14a3534" (UID: "b37263ad-40ef-4f70-a5fd-a2a1f14a3534"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 22:55:24.558069 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:24.558040    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b37263ad-40ef-4f70-a5fd-a2a1f14a3534-kube-api-access-h969q" pod "b37263ad-40ef-4f70-a5fd-a2a1f14a3534" (UID: "b37263ad-40ef-4f70-a5fd-a2a1f14a3534"). InnerVolumeSpecName "kube-api-access-h969q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:55:24.558193 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:24.558091    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b37263ad-40ef-4f70-a5fd-a2a1f14a3534-istio-token" pod "b37263ad-40ef-4f70-a5fd-a2a1f14a3534" (UID: "b37263ad-40ef-4f70-a5fd-a2a1f14a3534"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:55:24.656429 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:24.656397    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h969q\" (UniqueName: \"kubernetes.io/projected/b37263ad-40ef-4f70-a5fd-a2a1f14a3534-kube-api-access-h969q\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:55:24.656429 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:24.656424    4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b37263ad-40ef-4f70-a5fd-a2a1f14a3534-istio-token\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:55:24.656429 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:24.656434    4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b37263ad-40ef-4f70-a5fd-a2a1f14a3534-istio-envoy\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:55:24.656664 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:24.656443    4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b37263ad-40ef-4f70-a5fd-a2a1f14a3534-istio-podinfo\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:55:24.916495 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:24.916451    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:55:25.091993 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:25.091956    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:55:25.228700 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:25.228601    4431 generic.go:356] "Generic (PLEG): container finished" podID="b37263ad-40ef-4f70-a5fd-a2a1f14a3534" containerID="6fd3924b880e294f5fcf394f52a0a8a1cdfa65e442e5b58c253c58ca7471b272" exitCode=0
Nov 03 22:55:25.228700 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:25.228655    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5"
Nov 03 22:55:25.228700 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:25.228685    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5" event={"ID":"b37263ad-40ef-4f70-a5fd-a2a1f14a3534","Type":"ContainerDied","Data":"6fd3924b880e294f5fcf394f52a0a8a1cdfa65e442e5b58c253c58ca7471b272"}
Nov 03 22:55:25.228974 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:25.228740    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5" event={"ID":"b37263ad-40ef-4f70-a5fd-a2a1f14a3534","Type":"ContainerDied","Data":"6b84c339727786d6a49433098e9171fad6fb31e544d7722856644999c0b709b1"}
Nov 03 22:55:25.228974 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:25.228764    4431 scope.go:122] "RemoveContainer" containerID="6fd3924b880e294f5fcf394f52a0a8a1cdfa65e442e5b58c253c58ca7471b272"
Nov 03 22:55:25.240982 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:25.240923    4431 scope.go:122] "RemoveContainer" containerID="6fd3924b880e294f5fcf394f52a0a8a1cdfa65e442e5b58c253c58ca7471b272"
Nov 03 22:55:25.241274 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:55:25.241254    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fd3924b880e294f5fcf394f52a0a8a1cdfa65e442e5b58c253c58ca7471b272\": container with ID starting with 6fd3924b880e294f5fcf394f52a0a8a1cdfa65e442e5b58c253c58ca7471b272 not found: ID does not exist" containerID="6fd3924b880e294f5fcf394f52a0a8a1cdfa65e442e5b58c253c58ca7471b272"
Nov 03 22:55:25.241357 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:25.241287    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fd3924b880e294f5fcf394f52a0a8a1cdfa65e442e5b58c253c58ca7471b272"} err="failed to get container status \"6fd3924b880e294f5fcf394f52a0a8a1cdfa65e442e5b58c253c58ca7471b272\": rpc error: code = NotFound desc = could not find container \"6fd3924b880e294f5fcf394f52a0a8a1cdfa65e442e5b58c253c58ca7471b272\": container with ID starting with 6fd3924b880e294f5fcf394f52a0a8a1cdfa65e442e5b58c253c58ca7471b272 not found: ID does not exist"
Nov 03 22:55:25.292522 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:25.292490    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5"]
Nov 03 22:55:25.299162 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:25.299133    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-cflp5"]
Nov 03 22:55:25.555392 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:25.555305    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:55:25.862152 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:25.862061    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:55:26.006511 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:26.006475    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:55:26.092255 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:26.092202    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b37263ad-40ef-4f70-a5fd-a2a1f14a3534" path="/var/lib/kubelet/pods/b37263ad-40ef-4f70-a5fd-a2a1f14a3534/volumes"
Nov 03 22:55:26.576020 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:26.575981    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:55:28.239015 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:28.238979    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:55:29.152557 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:29.152493    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:55:29.318768 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:29.318732    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:55:29.494414 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:29.494387    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-963936-openshift-default-68b65f857c-phdvv"
Nov 03 22:55:29.628614 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:29.628582    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cbef074-f335-41d5-a69c-b84a958e87ee-istio-token\" (UniqueName: \"kubernetes.io/projected/2cbef074-f335-41d5-a69c-b84a958e87ee-istio-token\") pod \"2cbef074-f335-41d5-a69c-b84a958e87ee\" (UID: \"2cbef074-f335-41d5-a69c-b84a958e87ee\") "
Nov 03 22:55:29.628791 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:29.628676    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cbef074-f335-41d5-a69c-b84a958e87ee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cbef074-f335-41d5-a69c-b84a958e87ee-workload-certs\") pod \"2cbef074-f335-41d5-a69c-b84a958e87ee\" (UID: \"2cbef074-f335-41d5-a69c-b84a958e87ee\") "
Nov 03 22:55:29.628791 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:29.628746    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cbef074-f335-41d5-a69c-b84a958e87ee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cbef074-f335-41d5-a69c-b84a958e87ee-credential-socket\") pod \"2cbef074-f335-41d5-a69c-b84a958e87ee\" (UID: \"2cbef074-f335-41d5-a69c-b84a958e87ee\") "
Nov 03 22:55:29.628791 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:29.628783    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2cbef074-f335-41d5-a69c-b84a958e87ee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cbef074-f335-41d5-a69c-b84a958e87ee-istio-podinfo\") pod \"2cbef074-f335-41d5-a69c-b84a958e87ee\" (UID: \"2cbef074-f335-41d5-a69c-b84a958e87ee\") "
Nov 03 22:55:29.628950 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:29.628828    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cbef074-f335-41d5-a69c-b84a958e87ee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cbef074-f335-41d5-a69c-b84a958e87ee-istio-data\") pod \"2cbef074-f335-41d5-a69c-b84a958e87ee\" (UID: \"2cbef074-f335-41d5-a69c-b84a958e87ee\") "
Nov 03 22:55:29.628950 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:29.628857    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cbef074-f335-41d5-a69c-b84a958e87ee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cbef074-f335-41d5-a69c-b84a958e87ee-workload-socket\") pod \"2cbef074-f335-41d5-a69c-b84a958e87ee\" (UID: \"2cbef074-f335-41d5-a69c-b84a958e87ee\") "
Nov 03 22:55:29.628950 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:29.628892    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cbef074-f335-41d5-a69c-b84a958e87ee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cbef074-f335-41d5-a69c-b84a958e87ee-istiod-ca-cert\") pod \"2cbef074-f335-41d5-a69c-b84a958e87ee\" (UID: \"2cbef074-f335-41d5-a69c-b84a958e87ee\") "
Nov 03 22:55:29.629115 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:29.628974    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cbef074-f335-41d5-a69c-b84a958e87ee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cbef074-f335-41d5-a69c-b84a958e87ee-istio-envoy\") pod \"2cbef074-f335-41d5-a69c-b84a958e87ee\" (UID: \"2cbef074-f335-41d5-a69c-b84a958e87ee\") "
Nov 03 22:55:29.629115 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:29.629011    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cbef074-f335-41d5-a69c-b84a958e87ee-workload-certs" pod "2cbef074-f335-41d5-a69c-b84a958e87ee" (UID: "2cbef074-f335-41d5-a69c-b84a958e87ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:55:29.629115 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:29.629101    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cbef074-f335-41d5-a69c-b84a958e87ee-credential-socket" pod "2cbef074-f335-41d5-a69c-b84a958e87ee" (UID: "2cbef074-f335-41d5-a69c-b84a958e87ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:55:29.629318 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:29.629112    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cbef074-f335-41d5-a69c-b84a958e87ee-kube-api-access-nsnx9\" (UniqueName: \"kubernetes.io/projected/2cbef074-f335-41d5-a69c-b84a958e87ee-kube-api-access-nsnx9\") pod \"2cbef074-f335-41d5-a69c-b84a958e87ee\" (UID: \"2cbef074-f335-41d5-a69c-b84a958e87ee\") "
Nov 03 22:55:29.629318 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:29.629192    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cbef074-f335-41d5-a69c-b84a958e87ee-istio-data" pod "2cbef074-f335-41d5-a69c-b84a958e87ee" (UID: "2cbef074-f335-41d5-a69c-b84a958e87ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:55:29.629318 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:29.629204    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cbef074-f335-41d5-a69c-b84a958e87ee-workload-socket" pod "2cbef074-f335-41d5-a69c-b84a958e87ee" (UID: "2cbef074-f335-41d5-a69c-b84a958e87ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:55:29.629463 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:29.629446    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cbef074-f335-41d5-a69c-b84a958e87ee-istiod-ca-cert" pod "2cbef074-f335-41d5-a69c-b84a958e87ee" (UID: "2cbef074-f335-41d5-a69c-b84a958e87ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:55:29.629730 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:29.629708    4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cbef074-f335-41d5-a69c-b84a958e87ee-workload-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:55:29.629809 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:29.629734    4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cbef074-f335-41d5-a69c-b84a958e87ee-credential-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:55:29.629809 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:29.629748    4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cbef074-f335-41d5-a69c-b84a958e87ee-istio-data\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:55:29.629809 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:29.629762    4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cbef074-f335-41d5-a69c-b84a958e87ee-workload-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:55:29.629809 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:29.629774    4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cbef074-f335-41d5-a69c-b84a958e87ee-istiod-ca-cert\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:55:29.631158 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:29.631131    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cbef074-f335-41d5-a69c-b84a958e87ee-istio-envoy" pod "2cbef074-f335-41d5-a69c-b84a958e87ee" (UID: "2cbef074-f335-41d5-a69c-b84a958e87ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:55:29.631278 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:29.631264    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cbef074-f335-41d5-a69c-b84a958e87ee-kube-api-access-nsnx9" pod "2cbef074-f335-41d5-a69c-b84a958e87ee" (UID: "2cbef074-f335-41d5-a69c-b84a958e87ee"). InnerVolumeSpecName "kube-api-access-nsnx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:55:29.631428 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:29.631410    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cbef074-f335-41d5-a69c-b84a958e87ee-istio-token" pod "2cbef074-f335-41d5-a69c-b84a958e87ee" (UID: "2cbef074-f335-41d5-a69c-b84a958e87ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:55:29.631489 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:29.631409    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2cbef074-f335-41d5-a69c-b84a958e87ee-istio-podinfo" pod "2cbef074-f335-41d5-a69c-b84a958e87ee" (UID: "2cbef074-f335-41d5-a69c-b84a958e87ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 22:55:29.730748 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:29.730672    4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cbef074-f335-41d5-a69c-b84a958e87ee-istio-token\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:55:29.730748 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:29.730703    4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cbef074-f335-41d5-a69c-b84a958e87ee-istio-podinfo\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:55:29.730748 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:29.730714    4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cbef074-f335-41d5-a69c-b84a958e87ee-istio-envoy\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:55:29.730748 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:29.730722    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsnx9\" (UniqueName: \"kubernetes.io/projected/2cbef074-f335-41d5-a69c-b84a958e87ee-kube-api-access-nsnx9\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:55:30.010927 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:30.010826    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:55:30.274947 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:30.274868    4431 generic.go:356] "Generic (PLEG): container finished" podID="2cbef074-f335-41d5-a69c-b84a958e87ee" containerID="ca8359caacf7375447f51fd89147358bd7a1ecd1353d57c6ef1171690d006cf7" exitCode=0
Nov 03 22:55:30.274947 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:30.274903    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-963936-openshift-default-68b65f857c-phdvv" event={"ID":"2cbef074-f335-41d5-a69c-b84a958e87ee","Type":"ContainerDied","Data":"ca8359caacf7375447f51fd89147358bd7a1ecd1353d57c6ef1171690d006cf7"}
Nov 03 22:55:30.274947 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:30.274928    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-963936-openshift-default-68b65f857c-phdvv"
Nov 03 22:55:30.275246 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:30.274959    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-963936-openshift-default-68b65f857c-phdvv" event={"ID":"2cbef074-f335-41d5-a69c-b84a958e87ee","Type":"ContainerDied","Data":"9dd6dbf3831aa5ea22e88248af2bbdb00b85d345018f77597e9fbd4fbec65cd8"}
Nov 03 22:55:30.275246 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:30.274981    4431 scope.go:122] "RemoveContainer" containerID="ca8359caacf7375447f51fd89147358bd7a1ecd1353d57c6ef1171690d006cf7"
Nov 03 22:55:30.285489 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:30.285468    4431 scope.go:122] "RemoveContainer" containerID="12bd9f676d34dd68d6e917b86ac592cd27ca4690bd71c1ecf29d513985017952"
Nov 03 22:55:30.294979 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:30.294960    4431 scope.go:122] "RemoveContainer" containerID="ca8359caacf7375447f51fd89147358bd7a1ecd1353d57c6ef1171690d006cf7"
Nov 03 22:55:30.295273 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:55:30.295255    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca8359caacf7375447f51fd89147358bd7a1ecd1353d57c6ef1171690d006cf7\": container with ID starting with ca8359caacf7375447f51fd89147358bd7a1ecd1353d57c6ef1171690d006cf7 not found: ID does not exist" containerID="ca8359caacf7375447f51fd89147358bd7a1ecd1353d57c6ef1171690d006cf7"
Nov 03 22:55:30.295329 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:30.295281    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca8359caacf7375447f51fd89147358bd7a1ecd1353d57c6ef1171690d006cf7"} err="failed to get container status \"ca8359caacf7375447f51fd89147358bd7a1ecd1353d57c6ef1171690d006cf7\": rpc error: code = NotFound desc = could not find container \"ca8359caacf7375447f51fd89147358bd7a1ecd1353d57c6ef1171690d006cf7\": container with ID starting with ca8359caacf7375447f51fd89147358bd7a1ecd1353d57c6ef1171690d006cf7 not found: ID does not exist"
Nov 03 22:55:30.295329 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:30.295309    4431 scope.go:122] "RemoveContainer" containerID="12bd9f676d34dd68d6e917b86ac592cd27ca4690bd71c1ecf29d513985017952"
Nov 03 22:55:30.295531 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:55:30.295514    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12bd9f676d34dd68d6e917b86ac592cd27ca4690bd71c1ecf29d513985017952\": container with ID starting with 12bd9f676d34dd68d6e917b86ac592cd27ca4690bd71c1ecf29d513985017952 not found: ID does not exist" containerID="12bd9f676d34dd68d6e917b86ac592cd27ca4690bd71c1ecf29d513985017952"
Nov 03 22:55:30.295573 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:30.295535    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12bd9f676d34dd68d6e917b86ac592cd27ca4690bd71c1ecf29d513985017952"} err="failed to get container status \"12bd9f676d34dd68d6e917b86ac592cd27ca4690bd71c1ecf29d513985017952\": rpc error: code = NotFound desc = could not find container \"12bd9f676d34dd68d6e917b86ac592cd27ca4690bd71c1ecf29d513985017952\": container with ID starting with 12bd9f676d34dd68d6e917b86ac592cd27ca4690bd71c1ecf29d513985017952 not found: ID does not exist"
Nov 03 22:55:30.329322 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:30.329285    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-963936-openshift-default-68b65f857c-phdvv"]
Nov 03 22:55:30.341256 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:30.341230    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-963936-openshift-default-68b65f857c-phdvv"]
Nov 03 22:55:30.798322 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:30.798288    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:55:31.110598 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:31.110504    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:55:31.261377 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:55:31.261335    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:55:31.853799 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:31.853757    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:55:32.089338 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:32.089298    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cbef074-f335-41d5-a69c-b84a958e87ee" path="/var/lib/kubelet/pods/2cbef074-f335-41d5-a69c-b84a958e87ee/volumes"
Nov 03 22:55:33.449669 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:33.449627    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:55:33.923152 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:33.923114    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-281732/auth-test-model-281732-kserve-766d97949c-68cv5"]
Nov 03 22:55:34.008982 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:34.008941    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:55:35.158677 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:35.158645    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:55:36.875818 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:36.875785    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:55:38.533841 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:38.533809    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:55:40.120128 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:40.120044    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:55:41.724478 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:41.724435    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:55:43.278032 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:43.277999    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:55:44.814315 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:44.814283    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:55:46.311003 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:46.310971    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:55:47.847702 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:47.847669    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:55:48.946816 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:48.946776    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-281732/auth-test-model-281732-kserve-766d97949c-68cv5" podUID="7248574d-773d-4837-899b-d5d62166da9d" containerName="main" containerID="cri-o://cae888beef22ca52559cbe23e662d32dcc3ce67bcc02a9183d34e34e2231acfe" gracePeriod=45
Nov 03 22:55:49.011545 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:49.011507    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-281732/auth-test-model-281732-kserve-766d97949c-68cv5" podUID="7248574d-773d-4837-899b-d5d62166da9d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.138:8000/ready\": dial tcp 10.133.0.138:8000: connect: connection refused"
Nov 03 22:55:49.234545 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:49.234516    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-281732/auth-test-model-281732-kserve-766d97949c-68cv5"
Nov 03 22:55:49.290933 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:49.290902    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7248574d-773d-4837-899b-d5d62166da9d-tls-certs\" (UniqueName: \"kubernetes.io/secret/7248574d-773d-4837-899b-d5d62166da9d-tls-certs\") pod \"7248574d-773d-4837-899b-d5d62166da9d\" (UID: \"7248574d-773d-4837-899b-d5d62166da9d\") "
Nov 03 22:55:49.291138 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:49.291059    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7248574d-773d-4837-899b-d5d62166da9d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7248574d-773d-4837-899b-d5d62166da9d-model-cache\") pod \"7248574d-773d-4837-899b-d5d62166da9d\" (UID: \"7248574d-773d-4837-899b-d5d62166da9d\") "
Nov 03 22:55:49.291138 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:49.291109    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7248574d-773d-4837-899b-d5d62166da9d-home\" (UniqueName: \"kubernetes.io/empty-dir/7248574d-773d-4837-899b-d5d62166da9d-home\") pod \"7248574d-773d-4837-899b-d5d62166da9d\" (UID: \"7248574d-773d-4837-899b-d5d62166da9d\") "
Nov 03 22:55:49.291306 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:49.291144    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7248574d-773d-4837-899b-d5d62166da9d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7248574d-773d-4837-899b-d5d62166da9d-tmp-dir\") pod \"7248574d-773d-4837-899b-d5d62166da9d\" (UID: \"7248574d-773d-4837-899b-d5d62166da9d\") "
Nov 03 22:55:49.291306 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:49.291177    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7248574d-773d-4837-899b-d5d62166da9d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7248574d-773d-4837-899b-d5d62166da9d-dshm\") pod \"7248574d-773d-4837-899b-d5d62166da9d\" (UID: \"7248574d-773d-4837-899b-d5d62166da9d\") "
Nov 03 22:55:49.291306 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:49.291257    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7248574d-773d-4837-899b-d5d62166da9d-kube-api-access-bqzz6\" (UniqueName: \"kubernetes.io/projected/7248574d-773d-4837-899b-d5d62166da9d-kube-api-access-bqzz6\") pod \"7248574d-773d-4837-899b-d5d62166da9d\" (UID: \"7248574d-773d-4837-899b-d5d62166da9d\") "
Nov 03 22:55:49.291477 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:49.291381    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7248574d-773d-4837-899b-d5d62166da9d-model-cache" pod "7248574d-773d-4837-899b-d5d62166da9d" (UID: "7248574d-773d-4837-899b-d5d62166da9d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:55:49.291477 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:49.291418    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7248574d-773d-4837-899b-d5d62166da9d-tmp-dir" pod "7248574d-773d-4837-899b-d5d62166da9d" (UID: "7248574d-773d-4837-899b-d5d62166da9d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:55:49.291477 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:49.291454    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7248574d-773d-4837-899b-d5d62166da9d-home" pod "7248574d-773d-4837-899b-d5d62166da9d" (UID: "7248574d-773d-4837-899b-d5d62166da9d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:55:49.291899 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:49.291875    4431 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7248574d-773d-4837-899b-d5d62166da9d-model-cache\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:55:49.291899 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:49.291900    4431 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7248574d-773d-4837-899b-d5d62166da9d-home\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:55:49.292153 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:49.291913    4431 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7248574d-773d-4837-899b-d5d62166da9d-tmp-dir\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:55:49.293436 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:49.293410    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7248574d-773d-4837-899b-d5d62166da9d-tls-certs" pod "7248574d-773d-4837-899b-d5d62166da9d" (UID: "7248574d-773d-4837-899b-d5d62166da9d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:55:49.293596 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:49.293578    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7248574d-773d-4837-899b-d5d62166da9d-kube-api-access-bqzz6" pod "7248574d-773d-4837-899b-d5d62166da9d" (UID: "7248574d-773d-4837-899b-d5d62166da9d"). InnerVolumeSpecName "kube-api-access-bqzz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:55:49.293790 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:49.293772    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7248574d-773d-4837-899b-d5d62166da9d-dshm" pod "7248574d-773d-4837-899b-d5d62166da9d" (UID: "7248574d-773d-4837-899b-d5d62166da9d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:55:49.390447 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:49.390410    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:55:49.392850 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:49.392822    4431 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7248574d-773d-4837-899b-d5d62166da9d-dshm\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:55:49.392850 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:49.392849    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqzz6\" (UniqueName: \"kubernetes.io/projected/7248574d-773d-4837-899b-d5d62166da9d-kube-api-access-bqzz6\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:55:49.393013 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:49.392861    4431 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7248574d-773d-4837-899b-d5d62166da9d-tls-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:55:49.405203 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:49.405176    4431 generic.go:356] "Generic (PLEG): container finished" podID="7248574d-773d-4837-899b-d5d62166da9d" containerID="cae888beef22ca52559cbe23e662d32dcc3ce67bcc02a9183d34e34e2231acfe" exitCode=0
Nov 03 22:55:49.405359 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:49.405240    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-281732/auth-test-model-281732-kserve-766d97949c-68cv5" event={"ID":"7248574d-773d-4837-899b-d5d62166da9d","Type":"ContainerDied","Data":"cae888beef22ca52559cbe23e662d32dcc3ce67bcc02a9183d34e34e2231acfe"}
Nov 03 22:55:49.405359 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:49.405274    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-281732/auth-test-model-281732-kserve-766d97949c-68cv5" event={"ID":"7248574d-773d-4837-899b-d5d62166da9d","Type":"ContainerDied","Data":"11ca6e0e73482f6b489b58d6c84f2c50a8fe793707224b52552dc32c0a7a1c68"}
Nov 03 22:55:49.405359 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:49.405283    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-281732/auth-test-model-281732-kserve-766d97949c-68cv5"
Nov 03 22:55:49.405562 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:49.405290    4431 scope.go:122] "RemoveContainer" containerID="cae888beef22ca52559cbe23e662d32dcc3ce67bcc02a9183d34e34e2231acfe"
Nov 03 22:55:49.435371 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:49.434406    4431 scope.go:122] "RemoveContainer" containerID="cae888beef22ca52559cbe23e662d32dcc3ce67bcc02a9183d34e34e2231acfe"
Nov 03 22:55:49.436696 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:55:49.436664    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cae888beef22ca52559cbe23e662d32dcc3ce67bcc02a9183d34e34e2231acfe\": container with ID starting with cae888beef22ca52559cbe23e662d32dcc3ce67bcc02a9183d34e34e2231acfe not found: ID does not exist" containerID="cae888beef22ca52559cbe23e662d32dcc3ce67bcc02a9183d34e34e2231acfe"
Nov 03 22:55:49.436806 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:49.436699    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cae888beef22ca52559cbe23e662d32dcc3ce67bcc02a9183d34e34e2231acfe"} err="failed to get container status \"cae888beef22ca52559cbe23e662d32dcc3ce67bcc02a9183d34e34e2231acfe\": rpc error: code = NotFound desc = could not find container \"cae888beef22ca52559cbe23e662d32dcc3ce67bcc02a9183d34e34e2231acfe\": container with ID starting with cae888beef22ca52559cbe23e662d32dcc3ce67bcc02a9183d34e34e2231acfe not found: ID does not exist"
Nov 03 22:55:49.481860 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:49.481787    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-281732/auth-test-model-281732-kserve-766d97949c-68cv5"]
Nov 03 22:55:49.485757 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:49.485733    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-281732/auth-test-model-281732-kserve-766d97949c-68cv5"]
Nov 03 22:55:49.534432 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:49.534398    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:55:50.090984 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:50.090950    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7248574d-773d-4837-899b-d5d62166da9d" path="/var/lib/kubelet/pods/7248574d-773d-4837-899b-d5d62166da9d/volumes"
Nov 03 22:55:51.030032 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:51.029996    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:55:52.187790 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:52.187750    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-04988c/auth-test-model-04988c-kserve-78755cd7fc-k6tdb"]
Nov 03 22:55:52.301239 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:52.289781    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:55:52.732775 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:52.732742    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:55:52.933283 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:52.933251    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:55:53.654511 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:53.654471    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:55:53.810688 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:53.810652    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:55:54.538553 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:54.538505    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:55:54.677731 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:54.677694    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:55:56.250475 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:56.250438    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:55:57.476418 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:57.476382    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:55:57.647265 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:57.647209    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:55:58.104417 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:58.104386    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:55:59.149026 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:59.148947    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:55:59.226309 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:59.226272    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3920cb-675b9c8b6b-f2xtg"]
Nov 03 22:55:59.226885 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:59.226869    4431 cpu_manager.go:414] "containerMap: removing container" podUID="51131709-120f-4b00-a6ec-34e879ac168b" containerName="istio-proxy"
Nov 03 22:55:59.226935 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:59.226887    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51131709-120f-4b00-a6ec-34e879ac168b" containerName="istio-proxy"
Nov 03 22:55:59.226935 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:59.226900    4431 cpu_manager.go:414] "containerMap: removing container" podUID="2cbef074-f335-41d5-a69c-b84a958e87ee" containerName="istio-proxy"
Nov 03 22:55:59.226935 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:59.226905    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cbef074-f335-41d5-a69c-b84a958e87ee" containerName="istio-proxy"
Nov 03 22:55:59.226935 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:59.226922    4431 cpu_manager.go:414] "containerMap: removing container" podUID="b37263ad-40ef-4f70-a5fd-a2a1f14a3534" containerName="istio-proxy"
Nov 03 22:55:59.226935 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:59.226927    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b37263ad-40ef-4f70-a5fd-a2a1f14a3534" containerName="istio-proxy"
Nov 03 22:55:59.227146 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:59.226953    4431 cpu_manager.go:414] "containerMap: removing container" podUID="fffa2baf-eaca-4459-b8c4-63ded79ac8ec" containerName="payload-pre-processing"
Nov 03 22:55:59.227146 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:59.226958    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fffa2baf-eaca-4459-b8c4-63ded79ac8ec" containerName="payload-pre-processing"
Nov 03 22:55:59.227146 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:59.226970    4431 cpu_manager.go:414] "containerMap: removing container" podUID="7248574d-773d-4837-899b-d5d62166da9d" containerName="main"
Nov 03 22:55:59.227146 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:59.226974    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7248574d-773d-4837-899b-d5d62166da9d" containerName="main"
Nov 03 22:55:59.227146 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:59.227055    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="51131709-120f-4b00-a6ec-34e879ac168b" containerName="istio-proxy"
Nov 03 22:55:59.227146 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:59.227062    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cbef074-f335-41d5-a69c-b84a958e87ee" containerName="istio-proxy"
Nov 03 22:55:59.227146 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:59.227069    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="b37263ad-40ef-4f70-a5fd-a2a1f14a3534" containerName="istio-proxy"
Nov 03 22:55:59.227146 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:59.227081    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="fffa2baf-eaca-4459-b8c4-63ded79ac8ec" containerName="payload-pre-processing"
Nov 03 22:55:59.227146 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:59.227089    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="7248574d-773d-4837-899b-d5d62166da9d" containerName="main"
Nov 03 22:55:59.227146 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:59.227095    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="51131709-120f-4b00-a6ec-34e879ac168b" containerName="istio-proxy"
Nov 03 22:55:59.227146 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:59.227102    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cbef074-f335-41d5-a69c-b84a958e87ee" containerName="istio-proxy"
Nov 03 22:55:59.231762 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:59.231741    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3920cb-675b9c8b6b-f2xtg"
Nov 03 22:55:59.247176 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:59.247148    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-3920cb\""
Nov 03 22:55:59.262344 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:59.262313    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3920cb-675b9c8b6b-f2xtg"]
Nov 03 22:55:59.286828 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:59.286798    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3920cb-65745db9dc-czdrn"]
Nov 03 22:55:59.287404 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:59.287390    4431 cpu_manager.go:414] "containerMap: removing container" podUID="51131709-120f-4b00-a6ec-34e879ac168b" containerName="istio-proxy"
Nov 03 22:55:59.287477 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:59.287405    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51131709-120f-4b00-a6ec-34e879ac168b" containerName="istio-proxy"
Nov 03 22:55:59.287477 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:59.287419    4431 cpu_manager.go:414] "containerMap: removing container" podUID="2cbef074-f335-41d5-a69c-b84a958e87ee" containerName="istio-proxy"
Nov 03 22:55:59.287477 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:59.287425    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cbef074-f335-41d5-a69c-b84a958e87ee" containerName="istio-proxy"
Nov 03 22:55:59.291002 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:59.290986    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3920cb-65745db9dc-czdrn"
Nov 03 22:55:59.303208 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:59.303184    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:55:59.306761 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:59.306737    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-3920cb-dockercfg-hbhqj\""
Nov 03 22:55:59.311604 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:59.311577    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3920cb-65745db9dc-czdrn"]
Nov 03 22:55:59.327281 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:59.327252    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3d43443-d6a8-490e-8f57-391d7e53b373-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3920cb-65745db9dc-czdrn\" (UID: \"d3d43443-d6a8-490e-8f57-391d7e53b373\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3920cb-65745db9dc-czdrn"
Nov 03 22:55:59.327443 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:59.327426    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rcsc\" (UniqueName: \"kubernetes.io/projected/a6a0f3fc-cde3-4e0a-b6a0-dba3b775aa7b-kube-api-access-8rcsc\") pod \"payload-processing-e2e-ipp-a-3920cb-675b9c8b6b-f2xtg\" (UID: \"a6a0f3fc-cde3-4e0a-b6a0-dba3b775aa7b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3920cb-675b9c8b6b-f2xtg"
Nov 03 22:55:59.327548 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:59.327528    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gwqv\" (UniqueName: \"kubernetes.io/projected/d3d43443-d6a8-490e-8f57-391d7e53b373-kube-api-access-6gwqv\") pod \"payload-pre-processing-e2e-ipp-a-3920cb-65745db9dc-czdrn\" (UID: \"d3d43443-d6a8-490e-8f57-391d7e53b373\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3920cb-65745db9dc-czdrn"
Nov 03 22:55:59.327695 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:59.327669    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6a0f3fc-cde3-4e0a-b6a0-dba3b775aa7b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3920cb-675b9c8b6b-f2xtg\" (UID: \"a6a0f3fc-cde3-4e0a-b6a0-dba3b775aa7b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3920cb-675b9c8b6b-f2xtg"
Nov 03 22:55:59.429615 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:59.429578    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3d43443-d6a8-490e-8f57-391d7e53b373-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3920cb-65745db9dc-czdrn\" (UID: \"d3d43443-d6a8-490e-8f57-391d7e53b373\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3920cb-65745db9dc-czdrn"
Nov 03 22:55:59.429862 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:59.429722    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rcsc\" (UniqueName: \"kubernetes.io/projected/a6a0f3fc-cde3-4e0a-b6a0-dba3b775aa7b-kube-api-access-8rcsc\") pod \"payload-processing-e2e-ipp-a-3920cb-675b9c8b6b-f2xtg\" (UID: \"a6a0f3fc-cde3-4e0a-b6a0-dba3b775aa7b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3920cb-675b9c8b6b-f2xtg"
Nov 03 22:55:59.429862 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:59.429799    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gwqv\" (UniqueName: \"kubernetes.io/projected/d3d43443-d6a8-490e-8f57-391d7e53b373-kube-api-access-6gwqv\") pod \"payload-pre-processing-e2e-ipp-a-3920cb-65745db9dc-czdrn\" (UID: \"d3d43443-d6a8-490e-8f57-391d7e53b373\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3920cb-65745db9dc-czdrn"
Nov 03 22:55:59.430004 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:59.429879    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6a0f3fc-cde3-4e0a-b6a0-dba3b775aa7b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3920cb-675b9c8b6b-f2xtg\" (UID: \"a6a0f3fc-cde3-4e0a-b6a0-dba3b775aa7b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3920cb-675b9c8b6b-f2xtg"
Nov 03 22:55:59.430600 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:59.430570    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3d43443-d6a8-490e-8f57-391d7e53b373-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3920cb-65745db9dc-czdrn\" (UID: \"d3d43443-d6a8-490e-8f57-391d7e53b373\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3920cb-65745db9dc-czdrn"
Nov 03 22:55:59.430724 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:59.430700    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6a0f3fc-cde3-4e0a-b6a0-dba3b775aa7b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3920cb-675b9c8b6b-f2xtg\" (UID: \"a6a0f3fc-cde3-4e0a-b6a0-dba3b775aa7b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3920cb-675b9c8b6b-f2xtg"
Nov 03 22:55:59.453845 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:59.453816    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gwqv\" (UniqueName: \"kubernetes.io/projected/d3d43443-d6a8-490e-8f57-391d7e53b373-kube-api-access-6gwqv\") pod \"payload-pre-processing-e2e-ipp-a-3920cb-65745db9dc-czdrn\" (UID: \"d3d43443-d6a8-490e-8f57-391d7e53b373\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3920cb-65745db9dc-czdrn"
Nov 03 22:55:59.454002 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:59.453985    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rcsc\" (UniqueName: \"kubernetes.io/projected/a6a0f3fc-cde3-4e0a-b6a0-dba3b775aa7b-kube-api-access-8rcsc\") pod \"payload-processing-e2e-ipp-a-3920cb-675b9c8b6b-f2xtg\" (UID: \"a6a0f3fc-cde3-4e0a-b6a0-dba3b775aa7b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3920cb-675b9c8b6b-f2xtg"
Nov 03 22:55:59.546194 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:59.546160    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3920cb-675b9c8b6b-f2xtg"
Nov 03 22:55:59.605614 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:59.605586    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3920cb-65745db9dc-czdrn"
Nov 03 22:55:59.848671 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:59.848641    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3920cb-675b9c8b6b-f2xtg"]
Nov 03 22:55:59.850113 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:55:59.850083    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6a0f3fc_cde3_4e0a_b6a0_dba3b775aa7b.slice/crio-386f5ba9c803fbdbd2db8b724c610a9d6246f7a22d4b642047b040c575a667bb WatchSource:0}: Error finding container 386f5ba9c803fbdbd2db8b724c610a9d6246f7a22d4b642047b040c575a667bb: Status 404 returned error can't find the container with id 386f5ba9c803fbdbd2db8b724c610a9d6246f7a22d4b642047b040c575a667bb
Nov 03 22:55:59.937441 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:55:59.937409    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3920cb-65745db9dc-czdrn"]
Nov 03 22:55:59.939002 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:55:59.938975    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3d43443_d6a8_490e_8f57_391d7e53b373.slice/crio-2701334de17fa80e02df3dc568afba1010ae6132d7321ce3adf84fb3509422c4 WatchSource:0}: Error finding container 2701334de17fa80e02df3dc568afba1010ae6132d7321ce3adf84fb3509422c4: Status 404 returned error can't find the container with id 2701334de17fa80e02df3dc568afba1010ae6132d7321ce3adf84fb3509422c4
Nov 03 22:56:00.125582 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:00.125544    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:56:00.484794 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:00.484753    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3920cb-675b9c8b6b-f2xtg" event={"ID":"a6a0f3fc-cde3-4e0a-b6a0-dba3b775aa7b","Type":"ContainerStarted","Data":"7bdfae36ff97b795ad2818bbf1376a0ebb04b19a6452755c1aa00e46c9246e83"}
Nov 03 22:56:00.485226 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:00.484819    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3920cb-675b9c8b6b-f2xtg" event={"ID":"a6a0f3fc-cde3-4e0a-b6a0-dba3b775aa7b","Type":"ContainerStarted","Data":"386f5ba9c803fbdbd2db8b724c610a9d6246f7a22d4b642047b040c575a667bb"}
Nov 03 22:56:00.485226 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:00.484849    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3920cb-675b9c8b6b-f2xtg"
Nov 03 22:56:00.486612 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:00.486584    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3920cb-65745db9dc-czdrn" event={"ID":"d3d43443-d6a8-490e-8f57-391d7e53b373","Type":"ContainerStarted","Data":"ff2c4b120a1fb2a55a304e01eca6428b466584f9b022a7ae49b63b29e89aec05"}
Nov 03 22:56:00.486712 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:00.486632    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3920cb-65745db9dc-czdrn" event={"ID":"d3d43443-d6a8-490e-8f57-391d7e53b373","Type":"ContainerStarted","Data":"2701334de17fa80e02df3dc568afba1010ae6132d7321ce3adf84fb3509422c4"}
Nov 03 22:56:00.486712 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:00.486656    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3920cb-65745db9dc-czdrn"
Nov 03 22:56:00.525989 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:00.525920    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-3920cb-675b9c8b6b-f2xtg" podStartSLOduration=1.5259016669999999 podStartE2EDuration="1.525901667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:55:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:56:00.523730324 +0000 UTC m=+2584.549586178" watchObservedRunningTime="2026-11-03 22:56:00.525901667 +0000 UTC m=+2584.551757530"
Nov 03 22:56:00.565059 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:00.565008    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3920cb-65745db9dc-czdrn" podStartSLOduration=1.564995905 podStartE2EDuration="1.564995905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:55:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:56:00.562800831 +0000 UTC m=+2584.588656696" watchObservedRunningTime="2026-11-03 22:56:00.564995905 +0000 UTC m=+2584.590851767"
Nov 03 22:56:01.259904 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:56:01.259869    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:56:01.670957 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:01.670925    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:56:01.807994 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:01.807961    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:56:03.258356 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:03.258316    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:56:04.793433 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:04.793389    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:56:06.429276 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:06.429240    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:56:06.871188 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:06.871107    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-a940d5/sub-test-model-a940d5-kserve-656c9f7b4c-k7tz5"]
Nov 03 22:56:06.883769 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:06.883741    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-a940d5/sub-test-model-a940d5-kserve-656c9f7b4c-k7tz5"
Nov 03 22:56:06.903142 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:06.903109    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-a940d5/sub-test-model-a940d5-kserve-656c9f7b4c-k7tz5"]
Nov 03 22:56:06.909524 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:06.909493    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-a940d5\"/\"openshift-service-ca.crt\""
Nov 03 22:56:06.909680 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:06.909580    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-a940d5\"/\"kube-root-ca.crt\""
Nov 03 22:56:06.909753 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:06.909721    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-a940d5\"/\"default-dockercfg-ndg7t\""
Nov 03 22:56:06.912765 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:06.912742    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-a940d5\"/\"sub-test-model-a940d5-kserve-self-signed-certs\""
Nov 03 22:56:06.958879 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:06.958840    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ac36a59-81e9-4274-b40d-870331ee4c34-tmp-dir\") pod \"sub-test-model-a940d5-kserve-656c9f7b4c-k7tz5\" (UID: \"7ac36a59-81e9-4274-b40d-870331ee4c34\") " pod="ai-tenant-e2e-shared-b-w0-a940d5/sub-test-model-a940d5-kserve-656c9f7b4c-k7tz5"
Nov 03 22:56:06.959082 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:06.958985    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ac36a59-81e9-4274-b40d-870331ee4c34-tls-certs\") pod \"sub-test-model-a940d5-kserve-656c9f7b4c-k7tz5\" (UID: \"7ac36a59-81e9-4274-b40d-870331ee4c34\") " pod="ai-tenant-e2e-shared-b-w0-a940d5/sub-test-model-a940d5-kserve-656c9f7b4c-k7tz5"
Nov 03 22:56:06.959082 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:06.959063    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ac36a59-81e9-4274-b40d-870331ee4c34-home\") pod \"sub-test-model-a940d5-kserve-656c9f7b4c-k7tz5\" (UID: \"7ac36a59-81e9-4274-b40d-870331ee4c34\") " pod="ai-tenant-e2e-shared-b-w0-a940d5/sub-test-model-a940d5-kserve-656c9f7b4c-k7tz5"
Nov 03 22:56:06.959300 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:06.959272    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ac36a59-81e9-4274-b40d-870331ee4c34-model-cache\") pod \"sub-test-model-a940d5-kserve-656c9f7b4c-k7tz5\" (UID: \"7ac36a59-81e9-4274-b40d-870331ee4c34\") " pod="ai-tenant-e2e-shared-b-w0-a940d5/sub-test-model-a940d5-kserve-656c9f7b4c-k7tz5"
Nov 03 22:56:06.959414 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:06.959392    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45jgg\" (UniqueName: \"kubernetes.io/projected/7ac36a59-81e9-4274-b40d-870331ee4c34-kube-api-access-45jgg\") pod \"sub-test-model-a940d5-kserve-656c9f7b4c-k7tz5\" (UID: \"7ac36a59-81e9-4274-b40d-870331ee4c34\") " pod="ai-tenant-e2e-shared-b-w0-a940d5/sub-test-model-a940d5-kserve-656c9f7b4c-k7tz5"
Nov 03 22:56:06.959545 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:06.959529    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ac36a59-81e9-4274-b40d-870331ee4c34-dshm\") pod \"sub-test-model-a940d5-kserve-656c9f7b4c-k7tz5\" (UID: \"7ac36a59-81e9-4274-b40d-870331ee4c34\") " pod="ai-tenant-e2e-shared-b-w0-a940d5/sub-test-model-a940d5-kserve-656c9f7b4c-k7tz5"
Nov 03 22:56:06.976122 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:06.976092    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:56:07.060701 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:07.060662    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ac36a59-81e9-4274-b40d-870331ee4c34-dshm\") pod \"sub-test-model-a940d5-kserve-656c9f7b4c-k7tz5\" (UID: \"7ac36a59-81e9-4274-b40d-870331ee4c34\") " pod="ai-tenant-e2e-shared-b-w0-a940d5/sub-test-model-a940d5-kserve-656c9f7b4c-k7tz5"
Nov 03 22:56:07.060881 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:07.060719    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ac36a59-81e9-4274-b40d-870331ee4c34-tmp-dir\") pod \"sub-test-model-a940d5-kserve-656c9f7b4c-k7tz5\" (UID: \"7ac36a59-81e9-4274-b40d-870331ee4c34\") " pod="ai-tenant-e2e-shared-b-w0-a940d5/sub-test-model-a940d5-kserve-656c9f7b4c-k7tz5"
Nov 03 22:56:07.060881 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:07.060790    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ac36a59-81e9-4274-b40d-870331ee4c34-tls-certs\") pod \"sub-test-model-a940d5-kserve-656c9f7b4c-k7tz5\" (UID: \"7ac36a59-81e9-4274-b40d-870331ee4c34\") " pod="ai-tenant-e2e-shared-b-w0-a940d5/sub-test-model-a940d5-kserve-656c9f7b4c-k7tz5"
Nov 03 22:56:07.060881 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:07.060859    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ac36a59-81e9-4274-b40d-870331ee4c34-home\") pod \"sub-test-model-a940d5-kserve-656c9f7b4c-k7tz5\" (UID: \"7ac36a59-81e9-4274-b40d-870331ee4c34\") " pod="ai-tenant-e2e-shared-b-w0-a940d5/sub-test-model-a940d5-kserve-656c9f7b4c-k7tz5"
Nov 03 22:56:07.061036 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:07.060961    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ac36a59-81e9-4274-b40d-870331ee4c34-model-cache\") pod \"sub-test-model-a940d5-kserve-656c9f7b4c-k7tz5\" (UID: \"7ac36a59-81e9-4274-b40d-870331ee4c34\") " pod="ai-tenant-e2e-shared-b-w0-a940d5/sub-test-model-a940d5-kserve-656c9f7b4c-k7tz5"
Nov 03 22:56:07.061100 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:07.061034    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45jgg\" (UniqueName: \"kubernetes.io/projected/7ac36a59-81e9-4274-b40d-870331ee4c34-kube-api-access-45jgg\") pod \"sub-test-model-a940d5-kserve-656c9f7b4c-k7tz5\" (UID: \"7ac36a59-81e9-4274-b40d-870331ee4c34\") " pod="ai-tenant-e2e-shared-b-w0-a940d5/sub-test-model-a940d5-kserve-656c9f7b4c-k7tz5"
Nov 03 22:56:07.061258 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:07.061232    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ac36a59-81e9-4274-b40d-870331ee4c34-tmp-dir\") pod \"sub-test-model-a940d5-kserve-656c9f7b4c-k7tz5\" (UID: \"7ac36a59-81e9-4274-b40d-870331ee4c34\") " pod="ai-tenant-e2e-shared-b-w0-a940d5/sub-test-model-a940d5-kserve-656c9f7b4c-k7tz5"
Nov 03 22:56:07.061352 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:07.061285    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ac36a59-81e9-4274-b40d-870331ee4c34-home\") pod \"sub-test-model-a940d5-kserve-656c9f7b4c-k7tz5\" (UID: \"7ac36a59-81e9-4274-b40d-870331ee4c34\") " pod="ai-tenant-e2e-shared-b-w0-a940d5/sub-test-model-a940d5-kserve-656c9f7b4c-k7tz5"
Nov 03 22:56:07.061404 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:07.061354    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ac36a59-81e9-4274-b40d-870331ee4c34-model-cache\") pod \"sub-test-model-a940d5-kserve-656c9f7b4c-k7tz5\" (UID: \"7ac36a59-81e9-4274-b40d-870331ee4c34\") " pod="ai-tenant-e2e-shared-b-w0-a940d5/sub-test-model-a940d5-kserve-656c9f7b4c-k7tz5"
Nov 03 22:56:07.063173 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:07.063144    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ac36a59-81e9-4274-b40d-870331ee4c34-dshm\") pod \"sub-test-model-a940d5-kserve-656c9f7b4c-k7tz5\" (UID: \"7ac36a59-81e9-4274-b40d-870331ee4c34\") " pod="ai-tenant-e2e-shared-b-w0-a940d5/sub-test-model-a940d5-kserve-656c9f7b4c-k7tz5"
Nov 03 22:56:07.063844 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:07.063820    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ac36a59-81e9-4274-b40d-870331ee4c34-tls-certs\") pod \"sub-test-model-a940d5-kserve-656c9f7b4c-k7tz5\" (UID: \"7ac36a59-81e9-4274-b40d-870331ee4c34\") " pod="ai-tenant-e2e-shared-b-w0-a940d5/sub-test-model-a940d5-kserve-656c9f7b4c-k7tz5"
Nov 03 22:56:07.084708 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:07.084678    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45jgg\" (UniqueName: \"kubernetes.io/projected/7ac36a59-81e9-4274-b40d-870331ee4c34-kube-api-access-45jgg\") pod \"sub-test-model-a940d5-kserve-656c9f7b4c-k7tz5\" (UID: \"7ac36a59-81e9-4274-b40d-870331ee4c34\") " pod="ai-tenant-e2e-shared-b-w0-a940d5/sub-test-model-a940d5-kserve-656c9f7b4c-k7tz5"
Nov 03 22:56:07.209226 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:07.209178    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-a940d5/sub-test-model-a940d5-kserve-656c9f7b4c-k7tz5"
Nov 03 22:56:07.215763 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:07.215725    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-04988c/auth-test-model-04988c-kserve-78755cd7fc-k6tdb" podUID="bed3920d-932a-4e44-ae02-45aac5cbf633" containerName="main" containerID="cri-o://1c4b71ff880ff43e9d864940993befb193baef22419c0224f8f787989ef3b6b8" gracePeriod=45
Nov 03 22:56:07.330700 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:07.330303    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-04988c/auth-test-model-04988c-kserve-78755cd7fc-k6tdb" podUID="bed3920d-932a-4e44-ae02-45aac5cbf633" containerName="main" probeResult="failure" output="Get \"https://10.133.0.142:8000/ready\": dial tcp 10.133.0.142:8000: connect: connection refused"
Nov 03 22:56:07.479742 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:07.479711    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-04988c/auth-test-model-04988c-kserve-78755cd7fc-k6tdb"
Nov 03 22:56:07.530275 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:56:07.530246    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7ac36a59_81e9_4274_b40d_870331ee4c34.slice/crio-4871aafcb924f1b7c7bcc90ec9e664b8d9325297c30f0d67b1bbd8a603acd502 WatchSource:0}: Error finding container 4871aafcb924f1b7c7bcc90ec9e664b8d9325297c30f0d67b1bbd8a603acd502: Status 404 returned error can't find the container with id 4871aafcb924f1b7c7bcc90ec9e664b8d9325297c30f0d67b1bbd8a603acd502
Nov 03 22:56:07.538771 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:07.538740    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-a940d5/sub-test-model-a940d5-kserve-656c9f7b4c-k7tz5"]
Nov 03 22:56:07.544144 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:07.544113    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-a940d5/sub-test-model-a940d5-kserve-656c9f7b4c-k7tz5" event={"ID":"7ac36a59-81e9-4274-b40d-870331ee4c34","Type":"ContainerStarted","Data":"4871aafcb924f1b7c7bcc90ec9e664b8d9325297c30f0d67b1bbd8a603acd502"}
Nov 03 22:56:07.545985 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:07.545960    4431 generic.go:356] "Generic (PLEG): container finished" podID="bed3920d-932a-4e44-ae02-45aac5cbf633" containerID="1c4b71ff880ff43e9d864940993befb193baef22419c0224f8f787989ef3b6b8" exitCode=0
Nov 03 22:56:07.546109 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:07.546002    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-04988c/auth-test-model-04988c-kserve-78755cd7fc-k6tdb" event={"ID":"bed3920d-932a-4e44-ae02-45aac5cbf633","Type":"ContainerDied","Data":"1c4b71ff880ff43e9d864940993befb193baef22419c0224f8f787989ef3b6b8"}
Nov 03 22:56:07.546109 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:07.546025    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-04988c/auth-test-model-04988c-kserve-78755cd7fc-k6tdb"
Nov 03 22:56:07.546109 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:07.546044    4431 scope.go:122] "RemoveContainer" containerID="1c4b71ff880ff43e9d864940993befb193baef22419c0224f8f787989ef3b6b8"
Nov 03 22:56:07.546322 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:07.546032    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-04988c/auth-test-model-04988c-kserve-78755cd7fc-k6tdb" event={"ID":"bed3920d-932a-4e44-ae02-45aac5cbf633","Type":"ContainerDied","Data":"08a8f0ef6ffc238cca819386abdc678d0d1ee05e6697978596678715e57d89db"}
Nov 03 22:56:07.575929 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:07.574551    4431 scope.go:122] "RemoveContainer" containerID="1c4b71ff880ff43e9d864940993befb193baef22419c0224f8f787989ef3b6b8"
Nov 03 22:56:07.575929 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:56:07.574907    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c4b71ff880ff43e9d864940993befb193baef22419c0224f8f787989ef3b6b8\": container with ID starting with 1c4b71ff880ff43e9d864940993befb193baef22419c0224f8f787989ef3b6b8 not found: ID does not exist" containerID="1c4b71ff880ff43e9d864940993befb193baef22419c0224f8f787989ef3b6b8"
Nov 03 22:56:07.575929 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:07.574943    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c4b71ff880ff43e9d864940993befb193baef22419c0224f8f787989ef3b6b8"} err="failed to get container status \"1c4b71ff880ff43e9d864940993befb193baef22419c0224f8f787989ef3b6b8\": rpc error: code = NotFound desc = could not find container \"1c4b71ff880ff43e9d864940993befb193baef22419c0224f8f787989ef3b6b8\": container with ID starting with 1c4b71ff880ff43e9d864940993befb193baef22419c0224f8f787989ef3b6b8 not found: ID does not exist"
Nov 03 22:56:07.575929 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:07.574961    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bed3920d-932a-4e44-ae02-45aac5cbf633-kube-api-access-ztvsh\" (UniqueName: \"kubernetes.io/projected/bed3920d-932a-4e44-ae02-45aac5cbf633-kube-api-access-ztvsh\") pod \"bed3920d-932a-4e44-ae02-45aac5cbf633\" (UID: \"bed3920d-932a-4e44-ae02-45aac5cbf633\") "
Nov 03 22:56:07.575929 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:07.575059    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bed3920d-932a-4e44-ae02-45aac5cbf633-tls-certs\" (UniqueName: \"kubernetes.io/secret/bed3920d-932a-4e44-ae02-45aac5cbf633-tls-certs\") pod \"bed3920d-932a-4e44-ae02-45aac5cbf633\" (UID: \"bed3920d-932a-4e44-ae02-45aac5cbf633\") "
Nov 03 22:56:07.575929 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:07.575088    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bed3920d-932a-4e44-ae02-45aac5cbf633-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bed3920d-932a-4e44-ae02-45aac5cbf633-model-cache\") pod \"bed3920d-932a-4e44-ae02-45aac5cbf633\" (UID: \"bed3920d-932a-4e44-ae02-45aac5cbf633\") "
Nov 03 22:56:07.575929 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:07.575125    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bed3920d-932a-4e44-ae02-45aac5cbf633-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bed3920d-932a-4e44-ae02-45aac5cbf633-tmp-dir\") pod \"bed3920d-932a-4e44-ae02-45aac5cbf633\" (UID: \"bed3920d-932a-4e44-ae02-45aac5cbf633\") "
Nov 03 22:56:07.575929 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:07.575191    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bed3920d-932a-4e44-ae02-45aac5cbf633-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bed3920d-932a-4e44-ae02-45aac5cbf633-dshm\") pod \"bed3920d-932a-4e44-ae02-45aac5cbf633\" (UID: \"bed3920d-932a-4e44-ae02-45aac5cbf633\") "
Nov 03 22:56:07.575929 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:07.575236    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bed3920d-932a-4e44-ae02-45aac5cbf633-home\" (UniqueName: \"kubernetes.io/empty-dir/bed3920d-932a-4e44-ae02-45aac5cbf633-home\") pod \"bed3920d-932a-4e44-ae02-45aac5cbf633\" (UID: \"bed3920d-932a-4e44-ae02-45aac5cbf633\") "
Nov 03 22:56:07.575929 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:07.575451    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bed3920d-932a-4e44-ae02-45aac5cbf633-model-cache" pod "bed3920d-932a-4e44-ae02-45aac5cbf633" (UID: "bed3920d-932a-4e44-ae02-45aac5cbf633"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:56:07.575929 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:07.575833    4431 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bed3920d-932a-4e44-ae02-45aac5cbf633-model-cache\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:56:07.575929 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:07.575847    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bed3920d-932a-4e44-ae02-45aac5cbf633-tmp-dir" pod "bed3920d-932a-4e44-ae02-45aac5cbf633" (UID: "bed3920d-932a-4e44-ae02-45aac5cbf633"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:56:07.576494 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:07.576472    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bed3920d-932a-4e44-ae02-45aac5cbf633-home" pod "bed3920d-932a-4e44-ae02-45aac5cbf633" (UID: "bed3920d-932a-4e44-ae02-45aac5cbf633"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:56:07.578491 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:07.578470    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bed3920d-932a-4e44-ae02-45aac5cbf633-kube-api-access-ztvsh" pod "bed3920d-932a-4e44-ae02-45aac5cbf633" (UID: "bed3920d-932a-4e44-ae02-45aac5cbf633"). InnerVolumeSpecName "kube-api-access-ztvsh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:56:07.579529 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:07.579508    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bed3920d-932a-4e44-ae02-45aac5cbf633-tls-certs" pod "bed3920d-932a-4e44-ae02-45aac5cbf633" (UID: "bed3920d-932a-4e44-ae02-45aac5cbf633"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:56:07.581111 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:07.581085    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bed3920d-932a-4e44-ae02-45aac5cbf633-dshm" pod "bed3920d-932a-4e44-ae02-45aac5cbf633" (UID: "bed3920d-932a-4e44-ae02-45aac5cbf633"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:56:07.677037 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:07.676997    4431 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bed3920d-932a-4e44-ae02-45aac5cbf633-tls-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:56:07.677037 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:07.677034    4431 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bed3920d-932a-4e44-ae02-45aac5cbf633-tmp-dir\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:56:07.677037 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:07.677045    4431 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bed3920d-932a-4e44-ae02-45aac5cbf633-dshm\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:56:07.677290 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:07.677055    4431 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bed3920d-932a-4e44-ae02-45aac5cbf633-home\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:56:07.677290 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:07.677064    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztvsh\" (UniqueName: \"kubernetes.io/projected/bed3920d-932a-4e44-ae02-45aac5cbf633-kube-api-access-ztvsh\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:56:07.895401 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:07.895361    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-04988c/auth-test-model-04988c-kserve-78755cd7fc-k6tdb"]
Nov 03 22:56:07.901397 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:07.901368    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-04988c/auth-test-model-04988c-kserve-78755cd7fc-k6tdb"]
Nov 03 22:56:08.018671 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:08.018635    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:56:08.089079 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:08.089047    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bed3920d-932a-4e44-ae02-45aac5cbf633" path="/var/lib/kubelet/pods/bed3920d-932a-4e44-ae02-45aac5cbf633/volumes"
Nov 03 22:56:08.555142 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:08.555105    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-a940d5/sub-test-model-a940d5-kserve-656c9f7b4c-k7tz5" event={"ID":"7ac36a59-81e9-4274-b40d-870331ee4c34","Type":"ContainerStarted","Data":"346427fcd41cfc506c2760934837af5435cdfd89d382bea6eea0aaaabe85ccfc"}
Nov 03 22:56:08.598028 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:08.597974    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-a940d5/sub-test-model-a940d5-kserve-656c9f7b4c-k7tz5" podStartSLOduration=2.597959017 podStartE2EDuration="2.597959017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:56:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:56:08.595440376 +0000 UTC m=+2592.621296256" watchObservedRunningTime="2026-11-03 22:56:08.597959017 +0000 UTC m=+2592.623814880"
Nov 03 22:56:09.210153 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:09.210118    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-a940d5/sub-test-model-a940d5-kserve-656c9f7b4c-k7tz5"
Nov 03 22:56:09.477621 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:09.477530    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:56:10.977381 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:10.977344    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:56:11.496933 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:11.496905    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3920cb-65745db9dc-czdrn"
Nov 03 22:56:11.497106 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:11.496972    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3920cb-675b9c8b6b-f2xtg"
Nov 03 22:56:12.520317 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:12.520281    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:56:14.198041 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:14.198011    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:56:15.677266 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:15.677210    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:56:17.209549 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:17.209517    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-a940d5/sub-test-model-a940d5-kserve-656c9f7b4c-k7tz5"
Nov 03 22:56:17.222970 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:17.222938    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-a940d5/sub-test-model-a940d5-kserve-656c9f7b4c-k7tz5"
Nov 03 22:56:17.258765 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:17.258716    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:56:17.627406 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:17.627321    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-a940d5/sub-test-model-a940d5-kserve-656c9f7b4c-k7tz5"
Nov 03 22:56:18.813998 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:18.813966    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:56:20.409106 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:20.409060    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:56:21.864836 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:21.864800    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:56:23.384908 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:23.384868    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:56:24.914885 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:24.914848    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:56:26.708664 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:26.708624    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:56:28.215891 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:28.215852    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:56:29.707864 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:29.707830    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:56:29.848940 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:29.848889    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:56:29.989299 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:29.989170    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:56:31.261613 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:56:31.261576    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:56:31.308165 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:31.308129    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:56:32.906391 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:32.906356    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:56:34.602375 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:34.602336    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:56:36.210553 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:36.210521    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:56:37.910325 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:37.910253    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:56:39.584112 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:39.584077    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:56:40.625694 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:40.625661    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:56:40.766572 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:40.766541    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:56:41.153828 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:41.153791    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:56:42.745694 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:42.744836    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:56:44.267590 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:44.267554    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:56:45.857176 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:45.857139    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:56:47.410878 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:47.410845    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:56:48.886652 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:48.886618    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:56:49.510479 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:49.510441    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:56:50.350396 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:50.350361    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:56:51.913428 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:51.913394    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:56:53.404711 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:53.404676    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:56:54.647256 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:54.647197    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:56:55.140692 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:55.140662    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:56:56.615603 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:56.615566    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:56:58.108835 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:58.108795    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:56:59.587016 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:56:59.586977    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:57:01.151660 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:01.151625    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:57:01.260575 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:57:01.260533    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:57:02.701529 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:02.701492    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:57:04.234820 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:04.234784    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:57:05.838027 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:05.837992    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:57:07.504119 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:07.504080    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:57:08.869810 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:08.869771    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3920cb-675b9c8b6b-f2xtg"]
Nov 03 22:57:08.899793 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:08.899758    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3920cb-65745db9dc-czdrn"]
Nov 03 22:57:08.982769 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:08.982729    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:57:09.120796 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:09.120712    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:57:09.269078 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:09.269046    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:57:10.594493 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:10.594413    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:57:11.745105 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:11.745071    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:57:11.889115 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:11.889072    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:57:12.817943 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:12.817908    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:57:13.871000 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:13.870951    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-3920cb-675b9c8b6b-f2xtg" podUID="a6a0f3fc-cde3-4e0a-b6a0-dba3b775aa7b" containerName="payload-processing" containerID="cri-o://7bdfae36ff97b795ad2818bbf1376a0ebb04b19a6452755c1aa00e46c9246e83" gracePeriod=25
Nov 03 22:57:13.900769 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:13.900729    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3920cb-65745db9dc-czdrn" podUID="d3d43443-d6a8-490e-8f57-391d7e53b373" containerName="payload-pre-processing" containerID="cri-o://ff2c4b120a1fb2a55a304e01eca6428b466584f9b022a7ae49b63b29e89aec05" gracePeriod=25
Nov 03 22:57:14.183445 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:14.183418    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3920cb-675b9c8b6b-f2xtg"
Nov 03 22:57:14.228704 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:14.228676    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3920cb-65745db9dc-czdrn"
Nov 03 22:57:14.277517 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:14.277489    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6a0f3fc-cde3-4e0a-b6a0-dba3b775aa7b-kube-api-access-8rcsc\" (UniqueName: \"kubernetes.io/projected/a6a0f3fc-cde3-4e0a-b6a0-dba3b775aa7b-kube-api-access-8rcsc\") pod \"a6a0f3fc-cde3-4e0a-b6a0-dba3b775aa7b\" (UID: \"a6a0f3fc-cde3-4e0a-b6a0-dba3b775aa7b\") "
Nov 03 22:57:14.277681 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:14.277554    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6a0f3fc-cde3-4e0a-b6a0-dba3b775aa7b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6a0f3fc-cde3-4e0a-b6a0-dba3b775aa7b-plugins-config-volume\") pod \"a6a0f3fc-cde3-4e0a-b6a0-dba3b775aa7b\" (UID: \"a6a0f3fc-cde3-4e0a-b6a0-dba3b775aa7b\") "
Nov 03 22:57:14.277972 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:14.277946    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6a0f3fc-cde3-4e0a-b6a0-dba3b775aa7b-plugins-config-volume" pod "a6a0f3fc-cde3-4e0a-b6a0-dba3b775aa7b" (UID: "a6a0f3fc-cde3-4e0a-b6a0-dba3b775aa7b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:57:14.279727 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:14.279706    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6a0f3fc-cde3-4e0a-b6a0-dba3b775aa7b-kube-api-access-8rcsc" pod "a6a0f3fc-cde3-4e0a-b6a0-dba3b775aa7b" (UID: "a6a0f3fc-cde3-4e0a-b6a0-dba3b775aa7b"). InnerVolumeSpecName "kube-api-access-8rcsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:57:14.378840 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:14.378805    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3d43443-d6a8-490e-8f57-391d7e53b373-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3d43443-d6a8-490e-8f57-391d7e53b373-plugins-config-volume\") pod \"d3d43443-d6a8-490e-8f57-391d7e53b373\" (UID: \"d3d43443-d6a8-490e-8f57-391d7e53b373\") "
Nov 03 22:57:14.379042 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:14.378863    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3d43443-d6a8-490e-8f57-391d7e53b373-kube-api-access-6gwqv\" (UniqueName: \"kubernetes.io/projected/d3d43443-d6a8-490e-8f57-391d7e53b373-kube-api-access-6gwqv\") pod \"d3d43443-d6a8-490e-8f57-391d7e53b373\" (UID: \"d3d43443-d6a8-490e-8f57-391d7e53b373\") "
Nov 03 22:57:14.379284 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:14.379245    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3d43443-d6a8-490e-8f57-391d7e53b373-plugins-config-volume" pod "d3d43443-d6a8-490e-8f57-391d7e53b373" (UID: "d3d43443-d6a8-490e-8f57-391d7e53b373"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:57:14.379538 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:14.379520    4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3d43443-d6a8-490e-8f57-391d7e53b373-plugins-config-volume\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:57:14.379591 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:14.379541    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rcsc\" (UniqueName: \"kubernetes.io/projected/a6a0f3fc-cde3-4e0a-b6a0-dba3b775aa7b-kube-api-access-8rcsc\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:57:14.379591 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:14.379550    4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6a0f3fc-cde3-4e0a-b6a0-dba3b775aa7b-plugins-config-volume\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:57:14.381045 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:14.381022    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3d43443-d6a8-490e-8f57-391d7e53b373-kube-api-access-6gwqv" pod "d3d43443-d6a8-490e-8f57-391d7e53b373" (UID: "d3d43443-d6a8-490e-8f57-391d7e53b373"). InnerVolumeSpecName "kube-api-access-6gwqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:57:14.480720 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:14.480673    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gwqv\" (UniqueName: \"kubernetes.io/projected/d3d43443-d6a8-490e-8f57-391d7e53b373-kube-api-access-6gwqv\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:57:14.509840 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:14.509765    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:57:14.991886 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:14.991846    4431 generic.go:356] "Generic (PLEG): container finished" podID="a6a0f3fc-cde3-4e0a-b6a0-dba3b775aa7b" containerID="7bdfae36ff97b795ad2818bbf1376a0ebb04b19a6452755c1aa00e46c9246e83" exitCode=0
Nov 03 22:57:14.991886 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:14.991892    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3920cb-675b9c8b6b-f2xtg"
Nov 03 22:57:14.992402 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:14.991935    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3920cb-675b9c8b6b-f2xtg" event={"ID":"a6a0f3fc-cde3-4e0a-b6a0-dba3b775aa7b","Type":"ContainerDied","Data":"7bdfae36ff97b795ad2818bbf1376a0ebb04b19a6452755c1aa00e46c9246e83"}
Nov 03 22:57:14.992402 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:14.991993    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3920cb-675b9c8b6b-f2xtg" event={"ID":"a6a0f3fc-cde3-4e0a-b6a0-dba3b775aa7b","Type":"ContainerDied","Data":"386f5ba9c803fbdbd2db8b724c610a9d6246f7a22d4b642047b040c575a667bb"}
Nov 03 22:57:14.992402 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:14.992021    4431 scope.go:122] "RemoveContainer" containerID="7bdfae36ff97b795ad2818bbf1376a0ebb04b19a6452755c1aa00e46c9246e83"
Nov 03 22:57:14.993639 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:14.993613    4431 generic.go:356] "Generic (PLEG): container finished" podID="d3d43443-d6a8-490e-8f57-391d7e53b373" containerID="ff2c4b120a1fb2a55a304e01eca6428b466584f9b022a7ae49b63b29e89aec05" exitCode=0
Nov 03 22:57:14.993770 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:14.993664    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3920cb-65745db9dc-czdrn"
Nov 03 22:57:14.993770 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:14.993685    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3920cb-65745db9dc-czdrn" event={"ID":"d3d43443-d6a8-490e-8f57-391d7e53b373","Type":"ContainerDied","Data":"ff2c4b120a1fb2a55a304e01eca6428b466584f9b022a7ae49b63b29e89aec05"}
Nov 03 22:57:14.993770 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:14.993728    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3920cb-65745db9dc-czdrn" event={"ID":"d3d43443-d6a8-490e-8f57-391d7e53b373","Type":"ContainerDied","Data":"2701334de17fa80e02df3dc568afba1010ae6132d7321ce3adf84fb3509422c4"}
Nov 03 22:57:15.003745 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:15.003723    4431 scope.go:122] "RemoveContainer" containerID="7bdfae36ff97b795ad2818bbf1376a0ebb04b19a6452755c1aa00e46c9246e83"
Nov 03 22:57:15.004044 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:57:15.004022    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bdfae36ff97b795ad2818bbf1376a0ebb04b19a6452755c1aa00e46c9246e83\": container with ID starting with 7bdfae36ff97b795ad2818bbf1376a0ebb04b19a6452755c1aa00e46c9246e83 not found: ID does not exist" containerID="7bdfae36ff97b795ad2818bbf1376a0ebb04b19a6452755c1aa00e46c9246e83"
Nov 03 22:57:15.004128 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:15.004052    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bdfae36ff97b795ad2818bbf1376a0ebb04b19a6452755c1aa00e46c9246e83"} err="failed to get container status \"7bdfae36ff97b795ad2818bbf1376a0ebb04b19a6452755c1aa00e46c9246e83\": rpc error: code = NotFound desc = could not find container \"7bdfae36ff97b795ad2818bbf1376a0ebb04b19a6452755c1aa00e46c9246e83\": container with ID starting with 7bdfae36ff97b795ad2818bbf1376a0ebb04b19a6452755c1aa00e46c9246e83 not found: ID does not exist"
Nov 03 22:57:15.004128 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:15.004078    4431 scope.go:122] "RemoveContainer" containerID="ff2c4b120a1fb2a55a304e01eca6428b466584f9b022a7ae49b63b29e89aec05"
Nov 03 22:57:15.013083 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:15.013062    4431 scope.go:122] "RemoveContainer" containerID="ff2c4b120a1fb2a55a304e01eca6428b466584f9b022a7ae49b63b29e89aec05"
Nov 03 22:57:15.013344 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:57:15.013325    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff2c4b120a1fb2a55a304e01eca6428b466584f9b022a7ae49b63b29e89aec05\": container with ID starting with ff2c4b120a1fb2a55a304e01eca6428b466584f9b022a7ae49b63b29e89aec05 not found: ID does not exist" containerID="ff2c4b120a1fb2a55a304e01eca6428b466584f9b022a7ae49b63b29e89aec05"
Nov 03 22:57:15.013434 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:15.013354    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff2c4b120a1fb2a55a304e01eca6428b466584f9b022a7ae49b63b29e89aec05"} err="failed to get container status \"ff2c4b120a1fb2a55a304e01eca6428b466584f9b022a7ae49b63b29e89aec05\": rpc error: code = NotFound desc = could not find container \"ff2c4b120a1fb2a55a304e01eca6428b466584f9b022a7ae49b63b29e89aec05\": container with ID starting with ff2c4b120a1fb2a55a304e01eca6428b466584f9b022a7ae49b63b29e89aec05 not found: ID does not exist"
Nov 03 22:57:15.043359 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:15.043335    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3920cb-65745db9dc-czdrn"]
Nov 03 22:57:15.057694 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:15.057668    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3920cb-65745db9dc-czdrn"]
Nov 03 22:57:15.091601 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:15.091572    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3920cb-675b9c8b6b-f2xtg"]
Nov 03 22:57:15.097494 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:15.097469    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3920cb-675b9c8b6b-f2xtg"]
Nov 03 22:57:16.089693 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:16.089661    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6a0f3fc-cde3-4e0a-b6a0-dba3b775aa7b" path="/var/lib/kubelet/pods/a6a0f3fc-cde3-4e0a-b6a0-dba3b775aa7b/volumes"
Nov 03 22:57:16.090096 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:16.090082    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3d43443-d6a8-490e-8f57-391d7e53b373" path="/var/lib/kubelet/pods/d3d43443-d6a8-490e-8f57-391d7e53b373/volumes"
Nov 03 22:57:16.092630 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:16.092602    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:57:17.598850 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:17.598808    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:57:19.333327 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:19.333293    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:57:21.322147 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:21.322106    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:57:23.004625 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:23.004593    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:57:24.579109 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:24.579077    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:57:26.096135 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:26.096095    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:57:27.717682 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:27.717647    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:57:27.901561 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:27.901523    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:57:28.265427 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:28.265394    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2"]
Nov 03 22:57:28.266632 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:28.266596    4431 cpu_manager.go:414] "containerMap: removing container" podUID="a6a0f3fc-cde3-4e0a-b6a0-dba3b775aa7b" containerName="payload-processing"
Nov 03 22:57:28.266632 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:28.266630    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6a0f3fc-cde3-4e0a-b6a0-dba3b775aa7b" containerName="payload-processing"
Nov 03 22:57:28.266857 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:28.266686    4431 cpu_manager.go:414] "containerMap: removing container" podUID="bed3920d-932a-4e44-ae02-45aac5cbf633" containerName="main"
Nov 03 22:57:28.266857 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:28.266694    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bed3920d-932a-4e44-ae02-45aac5cbf633" containerName="main"
Nov 03 22:57:28.266857 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:28.266712    4431 cpu_manager.go:414] "containerMap: removing container" podUID="d3d43443-d6a8-490e-8f57-391d7e53b373" containerName="payload-pre-processing"
Nov 03 22:57:28.266857 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:28.266720    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3d43443-d6a8-490e-8f57-391d7e53b373" containerName="payload-pre-processing"
Nov 03 22:57:28.267085 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:28.266870    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3d43443-d6a8-490e-8f57-391d7e53b373" containerName="payload-pre-processing"
Nov 03 22:57:28.267085 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:28.266894    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6a0f3fc-cde3-4e0a-b6a0-dba3b775aa7b" containerName="payload-processing"
Nov 03 22:57:28.267085 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:28.266910    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="bed3920d-932a-4e44-ae02-45aac5cbf633" containerName="main"
Nov 03 22:57:28.271019 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:28.270995    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2"
Nov 03 22:57:28.291095 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:28.290987    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-ff80c0-openshift-default-dockercfg-v27gw\""
Nov 03 22:57:28.305043 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:28.304993    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2"]
Nov 03 22:57:28.352681 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:28.352640    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:57:28.400249 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:28.398261    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/681f7328-e1f5-4544-876d-410f30ee83e5-workload-certs\") pod \"e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2\" (UID: \"681f7328-e1f5-4544-876d-410f30ee83e5\") " pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2"
Nov 03 22:57:28.400249 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:28.398351    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/681f7328-e1f5-4544-876d-410f30ee83e5-istio-token\") pod \"e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2\" (UID: \"681f7328-e1f5-4544-876d-410f30ee83e5\") " pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2"
Nov 03 22:57:28.400249 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:28.398413    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/681f7328-e1f5-4544-876d-410f30ee83e5-credential-socket\") pod \"e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2\" (UID: \"681f7328-e1f5-4544-876d-410f30ee83e5\") " pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2"
Nov 03 22:57:28.400249 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:28.398450    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/681f7328-e1f5-4544-876d-410f30ee83e5-istiod-ca-cert\") pod \"e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2\" (UID: \"681f7328-e1f5-4544-876d-410f30ee83e5\") " pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2"
Nov 03 22:57:28.400249 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:28.398526    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/681f7328-e1f5-4544-876d-410f30ee83e5-istio-data\") pod \"e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2\" (UID: \"681f7328-e1f5-4544-876d-410f30ee83e5\") " pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2"
Nov 03 22:57:28.400249 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:28.398559    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/681f7328-e1f5-4544-876d-410f30ee83e5-istio-podinfo\") pod \"e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2\" (UID: \"681f7328-e1f5-4544-876d-410f30ee83e5\") " pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2"
Nov 03 22:57:28.400249 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:28.398620    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtnf5\" (UniqueName: \"kubernetes.io/projected/681f7328-e1f5-4544-876d-410f30ee83e5-kube-api-access-vtnf5\") pod \"e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2\" (UID: \"681f7328-e1f5-4544-876d-410f30ee83e5\") " pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2"
Nov 03 22:57:28.400249 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:28.398671    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/681f7328-e1f5-4544-876d-410f30ee83e5-workload-socket\") pod \"e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2\" (UID: \"681f7328-e1f5-4544-876d-410f30ee83e5\") " pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2"
Nov 03 22:57:28.400249 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:28.398702    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/681f7328-e1f5-4544-876d-410f30ee83e5-istio-envoy\") pod \"e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2\" (UID: \"681f7328-e1f5-4544-876d-410f30ee83e5\") " pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2"
Nov 03 22:57:28.499554 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:28.499514    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/681f7328-e1f5-4544-876d-410f30ee83e5-workload-certs\") pod \"e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2\" (UID: \"681f7328-e1f5-4544-876d-410f30ee83e5\") " pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2"
Nov 03 22:57:28.499753 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:28.499580    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/681f7328-e1f5-4544-876d-410f30ee83e5-istio-token\") pod \"e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2\" (UID: \"681f7328-e1f5-4544-876d-410f30ee83e5\") " pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2"
Nov 03 22:57:28.499753 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:28.499624    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/681f7328-e1f5-4544-876d-410f30ee83e5-credential-socket\") pod \"e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2\" (UID: \"681f7328-e1f5-4544-876d-410f30ee83e5\") " pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2"
Nov 03 22:57:28.499753 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:28.499647    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/681f7328-e1f5-4544-876d-410f30ee83e5-istiod-ca-cert\") pod \"e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2\" (UID: \"681f7328-e1f5-4544-876d-410f30ee83e5\") " pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2"
Nov 03 22:57:28.499753 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:28.499716    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/681f7328-e1f5-4544-876d-410f30ee83e5-istio-data\") pod \"e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2\" (UID: \"681f7328-e1f5-4544-876d-410f30ee83e5\") " pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2"
Nov 03 22:57:28.499753 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:28.499749    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/681f7328-e1f5-4544-876d-410f30ee83e5-istio-podinfo\") pod \"e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2\" (UID: \"681f7328-e1f5-4544-876d-410f30ee83e5\") " pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2"
Nov 03 22:57:28.500068 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:28.499806    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtnf5\" (UniqueName: \"kubernetes.io/projected/681f7328-e1f5-4544-876d-410f30ee83e5-kube-api-access-vtnf5\") pod \"e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2\" (UID: \"681f7328-e1f5-4544-876d-410f30ee83e5\") " pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2"
Nov 03 22:57:28.500068 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:28.499851    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/681f7328-e1f5-4544-876d-410f30ee83e5-workload-socket\") pod \"e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2\" (UID: \"681f7328-e1f5-4544-876d-410f30ee83e5\") " pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2"
Nov 03 22:57:28.500068 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:28.499881    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/681f7328-e1f5-4544-876d-410f30ee83e5-istio-envoy\") pod \"e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2\" (UID: \"681f7328-e1f5-4544-876d-410f30ee83e5\") " pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2"
Nov 03 22:57:28.500068 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:28.500025    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/681f7328-e1f5-4544-876d-410f30ee83e5-workload-certs\") pod \"e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2\" (UID: \"681f7328-e1f5-4544-876d-410f30ee83e5\") " pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2"
Nov 03 22:57:28.500068 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:28.500035    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/681f7328-e1f5-4544-876d-410f30ee83e5-credential-socket\") pod \"e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2\" (UID: \"681f7328-e1f5-4544-876d-410f30ee83e5\") " pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2"
Nov 03 22:57:28.500382 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:28.500350    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/681f7328-e1f5-4544-876d-410f30ee83e5-istio-data\") pod \"e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2\" (UID: \"681f7328-e1f5-4544-876d-410f30ee83e5\") " pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2"
Nov 03 22:57:28.500514 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:28.500489    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/681f7328-e1f5-4544-876d-410f30ee83e5-workload-socket\") pod \"e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2\" (UID: \"681f7328-e1f5-4544-876d-410f30ee83e5\") " pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2"
Nov 03 22:57:28.500585 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:28.500567    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/681f7328-e1f5-4544-876d-410f30ee83e5-istiod-ca-cert\") pod \"e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2\" (UID: \"681f7328-e1f5-4544-876d-410f30ee83e5\") " pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2"
Nov 03 22:57:28.502545 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:28.502512    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/681f7328-e1f5-4544-876d-410f30ee83e5-istio-envoy\") pod \"e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2\" (UID: \"681f7328-e1f5-4544-876d-410f30ee83e5\") " pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2"
Nov 03 22:57:28.502738 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:28.502716    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/681f7328-e1f5-4544-876d-410f30ee83e5-istio-podinfo\") pod \"e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2\" (UID: \"681f7328-e1f5-4544-876d-410f30ee83e5\") " pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2"
Nov 03 22:57:28.523354 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:28.523283    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/681f7328-e1f5-4544-876d-410f30ee83e5-istio-token\") pod \"e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2\" (UID: \"681f7328-e1f5-4544-876d-410f30ee83e5\") " pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2"
Nov 03 22:57:28.523354 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:28.523345    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtnf5\" (UniqueName: \"kubernetes.io/projected/681f7328-e1f5-4544-876d-410f30ee83e5-kube-api-access-vtnf5\") pod \"e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2\" (UID: \"681f7328-e1f5-4544-876d-410f30ee83e5\") " pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2"
Nov 03 22:57:28.594948 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:28.594906    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2"
Nov 03 22:57:28.712424 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:28.712384    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:57:28.926812 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:28.926768    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2"]
Nov 03 22:57:28.933439 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:28.933405    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:57:28.933564 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:28.933474    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:57:28.933564 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:28.933502    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:57:29.090758 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:29.090718    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2" event={"ID":"681f7328-e1f5-4544-876d-410f30ee83e5","Type":"ContainerStarted","Data":"56cd9f39de4e5a13cba87a2ef1dcb98947e79cdbcb8bc3c4e4731bb1c4248148"}
Nov 03 22:57:29.090949 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:29.090783    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2" event={"ID":"681f7328-e1f5-4544-876d-410f30ee83e5","Type":"ContainerStarted","Data":"0b79a8d0c0944b3ed0c9ce2bf75072dff1c26f569120e37528534fff53499c47"}
Nov 03 22:57:29.139305 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:29.139250    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2" podStartSLOduration=1.1392366090000001 podStartE2EDuration="1.139236609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:57:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:57:29.134877092 +0000 UTC m=+2673.160732954" watchObservedRunningTime="2026-11-03 22:57:29.139236609 +0000 UTC m=+2673.165092470"
Nov 03 22:57:29.310611 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:29.310520    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:57:29.595450 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:29.595371    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2"
Nov 03 22:57:30.603516 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:30.603480    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2"
Nov 03 22:57:31.029826 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:31.029793    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:57:31.102112 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:31.102079    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2"
Nov 03 22:57:31.103179 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:31.103156    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2"
Nov 03 22:57:31.261024 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:57:31.260980    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:57:31.592393 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:31.592342    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:57:31.745779 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:31.745745    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:57:33.114574 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:33.114523    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:57:34.959583 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:34.959540    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:57:34.982593 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:34.982553    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ff80c0-6794988c8d-w644b"]
Nov 03 22:57:35.027478 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:35.027417    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ff80c0-6794988c8d-w644b"]
Nov 03 22:57:35.027670 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:35.027540    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ff80c0-6794988c8d-w644b"
Nov 03 22:57:35.066236 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:35.042887    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-ff80c0\""
Nov 03 22:57:35.086506 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:35.068035    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-ff80c0-dockercfg-x7pz5\""
Nov 03 22:57:35.118853 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:35.118818    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4k5p\" (UniqueName: \"kubernetes.io/projected/5ce45ae5-a3a7-4df1-8052-30c8f0512cf6-kube-api-access-p4k5p\") pod \"payload-processing-e2e-ipp-b-ff80c0-6794988c8d-w644b\" (UID: \"5ce45ae5-a3a7-4df1-8052-30c8f0512cf6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ff80c0-6794988c8d-w644b"
Nov 03 22:57:35.119352 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:35.119320    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ce45ae5-a3a7-4df1-8052-30c8f0512cf6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ff80c0-6794988c8d-w644b\" (UID: \"5ce45ae5-a3a7-4df1-8052-30c8f0512cf6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ff80c0-6794988c8d-w644b"
Nov 03 22:57:35.220202 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:35.220106    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ce45ae5-a3a7-4df1-8052-30c8f0512cf6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ff80c0-6794988c8d-w644b\" (UID: \"5ce45ae5-a3a7-4df1-8052-30c8f0512cf6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ff80c0-6794988c8d-w644b"
Nov 03 22:57:35.220426 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:35.220275    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4k5p\" (UniqueName: \"kubernetes.io/projected/5ce45ae5-a3a7-4df1-8052-30c8f0512cf6-kube-api-access-p4k5p\") pod \"payload-processing-e2e-ipp-b-ff80c0-6794988c8d-w644b\" (UID: \"5ce45ae5-a3a7-4df1-8052-30c8f0512cf6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ff80c0-6794988c8d-w644b"
Nov 03 22:57:35.220969 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:35.220943    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ce45ae5-a3a7-4df1-8052-30c8f0512cf6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ff80c0-6794988c8d-w644b\" (UID: \"5ce45ae5-a3a7-4df1-8052-30c8f0512cf6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ff80c0-6794988c8d-w644b"
Nov 03 22:57:35.243798 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:35.243765    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4k5p\" (UniqueName: \"kubernetes.io/projected/5ce45ae5-a3a7-4df1-8052-30c8f0512cf6-kube-api-access-p4k5p\") pod \"payload-processing-e2e-ipp-b-ff80c0-6794988c8d-w644b\" (UID: \"5ce45ae5-a3a7-4df1-8052-30c8f0512cf6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ff80c0-6794988c8d-w644b"
Nov 03 22:57:35.396176 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:35.396139    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ff80c0-6794988c8d-w644b"
Nov 03 22:57:35.701485 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:35.701455    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ff80c0-6794988c8d-w644b"]
Nov 03 22:57:35.702717 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:57:35.702690    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ce45ae5_a3a7_4df1_8052_30c8f0512cf6.slice/crio-433508f2d9d8fc4f1428911ff495bbc30e12dff59f759b7b05fdc9d75e63046f WatchSource:0}: Error finding container 433508f2d9d8fc4f1428911ff495bbc30e12dff59f759b7b05fdc9d75e63046f: Status 404 returned error can't find the container with id 433508f2d9d8fc4f1428911ff495bbc30e12dff59f759b7b05fdc9d75e63046f
Nov 03 22:57:36.141325 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:36.141194    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ff80c0-6794988c8d-w644b" event={"ID":"5ce45ae5-a3a7-4df1-8052-30c8f0512cf6","Type":"ContainerStarted","Data":"518b6ebd709fa891c25dbd028c034cdcd42b90977e4894b00cc33e601721eecd"}
Nov 03 22:57:36.141325 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:36.141276    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ff80c0-6794988c8d-w644b" event={"ID":"5ce45ae5-a3a7-4df1-8052-30c8f0512cf6","Type":"ContainerStarted","Data":"433508f2d9d8fc4f1428911ff495bbc30e12dff59f759b7b05fdc9d75e63046f"}
Nov 03 22:57:36.141782 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:36.141393    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ff80c0-6794988c8d-w644b"
Nov 03 22:57:36.183782 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:36.183719    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-ff80c0-6794988c8d-w644b" podStartSLOduration=2.183699791 podStartE2EDuration="2.183699791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:57:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:57:36.180640182 +0000 UTC m=+2680.206496045" watchObservedRunningTime="2026-11-03 22:57:36.183699791 +0000 UTC m=+2680.209555654"
Nov 03 22:57:36.398629 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:36.398525    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:57:37.889580 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:37.889540    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:57:39.501366 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:39.501330    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:57:41.129187 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:41.129146    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:57:42.219934 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:42.219867    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:57:42.376845 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:42.376805    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:57:43.397757 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:43.397709    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:57:44.694457 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:44.694410    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:57:44.964474 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:44.964385    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:57:46.537335 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:46.537277    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:57:47.151742 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:47.151714    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ff80c0-6794988c8d-w644b"
Nov 03 22:57:48.156389 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:48.156352    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:57:49.749578 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:49.749541    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:57:50.340817 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:50.340777    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:57:51.917254 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:51.917198    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:57:53.419287 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:53.419239    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:57:54.769610 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:54.769579    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:57:54.986438 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:54.986390    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:57:56.438708 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:56.438674    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:57:56.583409 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:56.583378    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:57:56.875367 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:56.875287    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mx5q8_ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea/ovn-acl-logging/0.log"
Nov 03 22:57:56.883528 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:56.883502    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mx5q8_ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea/ovn-acl-logging/0.log"
Nov 03 22:57:57.036870 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:57.036823    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:57:57.198190 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:57.198155    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:57:58.254430 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:58.254396    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:57:59.831050 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:57:59.831008    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:58:00.823265 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:00.823229    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:58:00.972170 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:00.972136    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:58:01.259589 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:58:01.259554    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:58:01.679578 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:01.679546    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:58:03.068569 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:03.068539    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:58:03.144855 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:03.144824    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f48f46-7c74fffd4b-b86lk"]
Nov 03 22:58:03.150368 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:03.150347    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f48f46-7c74fffd4b-b86lk"
Nov 03 22:58:03.165240 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:03.165200    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f48f46-7c74fffd4b-b86lk"]
Nov 03 22:58:03.173520 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:03.173495    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-f48f46\""
Nov 03 22:58:03.207646 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:03.207609    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:58:03.213921 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:03.213895    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f48f46-9875d54dc-m8hbh"]
Nov 03 22:58:03.218468 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:03.218446    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f48f46-9875d54dc-m8hbh"
Nov 03 22:58:03.226138 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:03.226091    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzc7c\" (UniqueName: \"kubernetes.io/projected/3fe7c171-abe2-428d-81e1-d13da5f0c376-kube-api-access-vzc7c\") pod \"payload-processing-e2e-ipp-b-f48f46-7c74fffd4b-b86lk\" (UID: \"3fe7c171-abe2-428d-81e1-d13da5f0c376\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f48f46-7c74fffd4b-b86lk"
Nov 03 22:58:03.226253 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:03.226175    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fe7c171-abe2-428d-81e1-d13da5f0c376-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f48f46-7c74fffd4b-b86lk\" (UID: \"3fe7c171-abe2-428d-81e1-d13da5f0c376\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f48f46-7c74fffd4b-b86lk"
Nov 03 22:58:03.251261 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:03.246542    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f48f46-9875d54dc-m8hbh"]
Nov 03 22:58:03.259511 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:03.259480    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-f48f46-dockercfg-fc5hx\""
Nov 03 22:58:03.328775 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:03.328659    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgwlt\" (UniqueName: \"kubernetes.io/projected/605d2d22-11a8-420e-9776-f4d173819a04-kube-api-access-dgwlt\") pod \"payload-pre-processing-e2e-ipp-b-f48f46-9875d54dc-m8hbh\" (UID: \"605d2d22-11a8-420e-9776-f4d173819a04\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f48f46-9875d54dc-m8hbh"
Nov 03 22:58:03.328948 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:03.328910    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzc7c\" (UniqueName: \"kubernetes.io/projected/3fe7c171-abe2-428d-81e1-d13da5f0c376-kube-api-access-vzc7c\") pod \"payload-processing-e2e-ipp-b-f48f46-7c74fffd4b-b86lk\" (UID: \"3fe7c171-abe2-428d-81e1-d13da5f0c376\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f48f46-7c74fffd4b-b86lk"
Nov 03 22:58:03.329047 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:03.329026    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fe7c171-abe2-428d-81e1-d13da5f0c376-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f48f46-7c74fffd4b-b86lk\" (UID: \"3fe7c171-abe2-428d-81e1-d13da5f0c376\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f48f46-7c74fffd4b-b86lk"
Nov 03 22:58:03.329120 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:03.329083    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/605d2d22-11a8-420e-9776-f4d173819a04-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f48f46-9875d54dc-m8hbh\" (UID: \"605d2d22-11a8-420e-9776-f4d173819a04\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f48f46-9875d54dc-m8hbh"
Nov 03 22:58:03.330074 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:03.330040    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fe7c171-abe2-428d-81e1-d13da5f0c376-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f48f46-7c74fffd4b-b86lk\" (UID: \"3fe7c171-abe2-428d-81e1-d13da5f0c376\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f48f46-7c74fffd4b-b86lk"
Nov 03 22:58:03.363186 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:03.363148    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:58:03.390769 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:03.384978    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzc7c\" (UniqueName: \"kubernetes.io/projected/3fe7c171-abe2-428d-81e1-d13da5f0c376-kube-api-access-vzc7c\") pod \"payload-processing-e2e-ipp-b-f48f46-7c74fffd4b-b86lk\" (UID: \"3fe7c171-abe2-428d-81e1-d13da5f0c376\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f48f46-7c74fffd4b-b86lk"
Nov 03 22:58:03.431461 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:03.431419    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgwlt\" (UniqueName: \"kubernetes.io/projected/605d2d22-11a8-420e-9776-f4d173819a04-kube-api-access-dgwlt\") pod \"payload-pre-processing-e2e-ipp-b-f48f46-9875d54dc-m8hbh\" (UID: \"605d2d22-11a8-420e-9776-f4d173819a04\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f48f46-9875d54dc-m8hbh"
Nov 03 22:58:03.431680 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:03.431592    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/605d2d22-11a8-420e-9776-f4d173819a04-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f48f46-9875d54dc-m8hbh\" (UID: \"605d2d22-11a8-420e-9776-f4d173819a04\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f48f46-9875d54dc-m8hbh"
Nov 03 22:58:03.432423 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:03.432399    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/605d2d22-11a8-420e-9776-f4d173819a04-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f48f46-9875d54dc-m8hbh\" (UID: \"605d2d22-11a8-420e-9776-f4d173819a04\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f48f46-9875d54dc-m8hbh"
Nov 03 22:58:03.458535 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:03.458505    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgwlt\" (UniqueName: \"kubernetes.io/projected/605d2d22-11a8-420e-9776-f4d173819a04-kube-api-access-dgwlt\") pod \"payload-pre-processing-e2e-ipp-b-f48f46-9875d54dc-m8hbh\" (UID: \"605d2d22-11a8-420e-9776-f4d173819a04\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f48f46-9875d54dc-m8hbh"
Nov 03 22:58:03.465386 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:03.465359    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f48f46-7c74fffd4b-b86lk"
Nov 03 22:58:03.545695 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:03.545450    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f48f46-9875d54dc-m8hbh"
Nov 03 22:58:03.779162 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:03.779136    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f48f46-7c74fffd4b-b86lk"]
Nov 03 22:58:03.781180 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:58:03.781152    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fe7c171_abe2_428d_81e1_d13da5f0c376.slice/crio-b168fbaf18bc09ebf6d222d7b556413d40ad9cffd85aff74b6312a0451b4d401 WatchSource:0}: Error finding container b168fbaf18bc09ebf6d222d7b556413d40ad9cffd85aff74b6312a0451b4d401: Status 404 returned error can't find the container with id b168fbaf18bc09ebf6d222d7b556413d40ad9cffd85aff74b6312a0451b4d401
Nov 03 22:58:03.854649 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:03.854619    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f48f46-9875d54dc-m8hbh"]
Nov 03 22:58:03.855495 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:58:03.855470    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod605d2d22_11a8_420e_9776_f4d173819a04.slice/crio-4cded80e934f3dd56412f910c8727731c6d7e11a279cdcc6c6ae811595ff4f0d WatchSource:0}: Error finding container 4cded80e934f3dd56412f910c8727731c6d7e11a279cdcc6c6ae811595ff4f0d: Status 404 returned error can't find the container with id 4cded80e934f3dd56412f910c8727731c6d7e11a279cdcc6c6ae811595ff4f0d
Nov 03 22:58:04.412911 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:04.412872    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f48f46-9875d54dc-m8hbh" event={"ID":"605d2d22-11a8-420e-9776-f4d173819a04","Type":"ContainerStarted","Data":"9915d1ed9660ace2cda9e402987737878e704963d2e535703226bc5376b0f07d"}
Nov 03 22:58:04.413389 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:04.412937    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f48f46-9875d54dc-m8hbh" event={"ID":"605d2d22-11a8-420e-9776-f4d173819a04","Type":"ContainerStarted","Data":"4cded80e934f3dd56412f910c8727731c6d7e11a279cdcc6c6ae811595ff4f0d"}
Nov 03 22:58:04.413389 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:04.413074    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f48f46-9875d54dc-m8hbh"
Nov 03 22:58:04.414449 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:04.414428    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f48f46-7c74fffd4b-b86lk" event={"ID":"3fe7c171-abe2-428d-81e1-d13da5f0c376","Type":"ContainerStarted","Data":"9a0337b91aaa74144c37601fccdb09507ade8eba9247b7a2b1d1db9a3aba9541"}
Nov 03 22:58:04.414571 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:04.414462    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f48f46-7c74fffd4b-b86lk" event={"ID":"3fe7c171-abe2-428d-81e1-d13da5f0c376","Type":"ContainerStarted","Data":"b168fbaf18bc09ebf6d222d7b556413d40ad9cffd85aff74b6312a0451b4d401"}
Nov 03 22:58:04.414571 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:04.414560    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f48f46-7c74fffd4b-b86lk"
Nov 03 22:58:04.452770 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:04.452724    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f48f46-9875d54dc-m8hbh" podStartSLOduration=1.452707287 podStartE2EDuration="1.452707287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:58:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:58:04.452091442 +0000 UTC m=+2708.477947306" watchObservedRunningTime="2026-11-03 22:58:04.452707287 +0000 UTC m=+2708.478563157"
Nov 03 22:58:04.493493 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:04.493405    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-f48f46-7c74fffd4b-b86lk" podStartSLOduration=1.493386592 podStartE2EDuration="1.493386592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:58:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:58:04.491131343 +0000 UTC m=+2708.516987186" watchObservedRunningTime="2026-11-03 22:58:04.493386592 +0000 UTC m=+2708.519242456"
Nov 03 22:58:04.943967 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:04.943937    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:58:06.515827 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:06.515768    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:58:08.050985 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:08.050952    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:58:08.196115 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:08.196084    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:58:09.786241 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:09.784899    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:58:11.407161 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:11.407122    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:58:12.566874 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:12.566829    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:58:13.000623 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:13.000586    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:58:13.201616 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:13.201571    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:58:13.717801 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:13.717733    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:58:13.873244 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:13.873181    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:58:14.610227 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:14.610188    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:58:15.424989 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:15.424958    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f48f46-7c74fffd4b-b86lk"
Nov 03 22:58:15.425407 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:15.425032    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f48f46-9875d54dc-m8hbh"
Nov 03 22:58:16.337171 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:16.337127    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:58:17.768627 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:17.768592    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:58:17.941237 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:17.937151    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:58:19.534102 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:19.534056    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:58:21.141803 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:21.141767    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:58:21.220291 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:21.220250    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-15e7a7-79b4b8dbd4-pb92m"]
Nov 03 22:58:21.225951 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:21.225933    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-15e7a7-79b4b8dbd4-pb92m"
Nov 03 22:58:21.242612 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:21.242583    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-15e7a7-79b4b8dbd4-pb92m"]
Nov 03 22:58:21.249717 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:21.249689    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-15e7a7\""
Nov 03 22:58:21.249869 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:21.249775    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-15e7a7-dockercfg-x5tfn\""
Nov 03 22:58:21.284526 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:21.284494    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:58:21.289386 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:21.289356    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxnkt\" (UniqueName: \"kubernetes.io/projected/cfb72051-95d9-406f-b200-e676af52c8ab-kube-api-access-pxnkt\") pod \"payload-processing-e2e-ipp-a-15e7a7-79b4b8dbd4-pb92m\" (UID: \"cfb72051-95d9-406f-b200-e676af52c8ab\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-15e7a7-79b4b8dbd4-pb92m"
Nov 03 22:58:21.289540 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:21.289512    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfb72051-95d9-406f-b200-e676af52c8ab-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-15e7a7-79b4b8dbd4-pb92m\" (UID: \"cfb72051-95d9-406f-b200-e676af52c8ab\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-15e7a7-79b4b8dbd4-pb92m"
Nov 03 22:58:21.289540 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:21.289520    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-15e7a7-5467566b7-npkkl"]
Nov 03 22:58:21.294012 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:21.293993    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15e7a7-5467566b7-npkkl"
Nov 03 22:58:21.310865 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:21.310835    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-15e7a7-5467566b7-npkkl"]
Nov 03 22:58:21.403289 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:21.403188    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxnkt\" (UniqueName: \"kubernetes.io/projected/cfb72051-95d9-406f-b200-e676af52c8ab-kube-api-access-pxnkt\") pod \"payload-processing-e2e-ipp-a-15e7a7-79b4b8dbd4-pb92m\" (UID: \"cfb72051-95d9-406f-b200-e676af52c8ab\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-15e7a7-79b4b8dbd4-pb92m"
Nov 03 22:58:21.403632 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:21.403601    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2a90093-86e4-4bed-9892-31150b35ff48-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-15e7a7-5467566b7-npkkl\" (UID: \"e2a90093-86e4-4bed-9892-31150b35ff48\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15e7a7-5467566b7-npkkl"
Nov 03 22:58:21.403822 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:21.403798    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgl99\" (UniqueName: \"kubernetes.io/projected/e2a90093-86e4-4bed-9892-31150b35ff48-kube-api-access-vgl99\") pod \"payload-pre-processing-e2e-ipp-a-15e7a7-5467566b7-npkkl\" (UID: \"e2a90093-86e4-4bed-9892-31150b35ff48\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15e7a7-5467566b7-npkkl"
Nov 03 22:58:21.403949 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:21.403932    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfb72051-95d9-406f-b200-e676af52c8ab-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-15e7a7-79b4b8dbd4-pb92m\" (UID: \"cfb72051-95d9-406f-b200-e676af52c8ab\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-15e7a7-79b4b8dbd4-pb92m"
Nov 03 22:58:21.405031 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:21.404993    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfb72051-95d9-406f-b200-e676af52c8ab-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-15e7a7-79b4b8dbd4-pb92m\" (UID: \"cfb72051-95d9-406f-b200-e676af52c8ab\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-15e7a7-79b4b8dbd4-pb92m"
Nov 03 22:58:21.432869 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:21.432837    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxnkt\" (UniqueName: \"kubernetes.io/projected/cfb72051-95d9-406f-b200-e676af52c8ab-kube-api-access-pxnkt\") pod \"payload-processing-e2e-ipp-a-15e7a7-79b4b8dbd4-pb92m\" (UID: \"cfb72051-95d9-406f-b200-e676af52c8ab\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-15e7a7-79b4b8dbd4-pb92m"
Nov 03 22:58:21.505842 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:21.505807    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2a90093-86e4-4bed-9892-31150b35ff48-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-15e7a7-5467566b7-npkkl\" (UID: \"e2a90093-86e4-4bed-9892-31150b35ff48\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15e7a7-5467566b7-npkkl"
Nov 03 22:58:21.506028 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:21.505864    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgl99\" (UniqueName: \"kubernetes.io/projected/e2a90093-86e4-4bed-9892-31150b35ff48-kube-api-access-vgl99\") pod \"payload-pre-processing-e2e-ipp-a-15e7a7-5467566b7-npkkl\" (UID: \"e2a90093-86e4-4bed-9892-31150b35ff48\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15e7a7-5467566b7-npkkl"
Nov 03 22:58:21.506601 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:21.506581    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2a90093-86e4-4bed-9892-31150b35ff48-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-15e7a7-5467566b7-npkkl\" (UID: \"e2a90093-86e4-4bed-9892-31150b35ff48\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15e7a7-5467566b7-npkkl"
Nov 03 22:58:21.529459 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:21.529431    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgl99\" (UniqueName: \"kubernetes.io/projected/e2a90093-86e4-4bed-9892-31150b35ff48-kube-api-access-vgl99\") pod \"payload-pre-processing-e2e-ipp-a-15e7a7-5467566b7-npkkl\" (UID: \"e2a90093-86e4-4bed-9892-31150b35ff48\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15e7a7-5467566b7-npkkl"
Nov 03 22:58:21.545424 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:21.545398    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-15e7a7-79b4b8dbd4-pb92m"
Nov 03 22:58:21.560690 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:21.560661    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:58:21.607842 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:21.607807    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15e7a7-5467566b7-npkkl"
Nov 03 22:58:21.877970 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:21.877935    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-15e7a7-79b4b8dbd4-pb92m"]
Nov 03 22:58:21.878817 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:58:21.878789    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfb72051_95d9_406f_b200_e676af52c8ab.slice/crio-2bdd7e4efa77c8c47bf35029d24c7b24216710e5ee2fa20a0ebe4a59aa96e640 WatchSource:0}: Error finding container 2bdd7e4efa77c8c47bf35029d24c7b24216710e5ee2fa20a0ebe4a59aa96e640: Status 404 returned error can't find the container with id 2bdd7e4efa77c8c47bf35029d24c7b24216710e5ee2fa20a0ebe4a59aa96e640
Nov 03 22:58:21.931341 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:21.931318    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-15e7a7-5467566b7-npkkl"]
Nov 03 22:58:21.931940 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:58:21.931911    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2a90093_86e4_4bed_9892_31150b35ff48.slice/crio-b8b803690ea3aa6237e8f689d77e7d7850b9566c1ae96d25e8a326a47ff2b169 WatchSource:0}: Error finding container b8b803690ea3aa6237e8f689d77e7d7850b9566c1ae96d25e8a326a47ff2b169: Status 404 returned error can't find the container with id b8b803690ea3aa6237e8f689d77e7d7850b9566c1ae96d25e8a326a47ff2b169
Nov 03 22:58:22.562454 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:22.562410    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15e7a7-5467566b7-npkkl" event={"ID":"e2a90093-86e4-4bed-9892-31150b35ff48","Type":"ContainerStarted","Data":"0dc27536e80b98489f5e1dafd8154c7feae26013a8b445c2af60a2cf9fcc657d"}
Nov 03 22:58:22.562873 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:22.562486    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15e7a7-5467566b7-npkkl"
Nov 03 22:58:22.562873 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:22.562517    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15e7a7-5467566b7-npkkl" event={"ID":"e2a90093-86e4-4bed-9892-31150b35ff48","Type":"ContainerStarted","Data":"b8b803690ea3aa6237e8f689d77e7d7850b9566c1ae96d25e8a326a47ff2b169"}
Nov 03 22:58:22.564163 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:22.564136    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-15e7a7-79b4b8dbd4-pb92m" event={"ID":"cfb72051-95d9-406f-b200-e676af52c8ab","Type":"ContainerStarted","Data":"08a2498654efb31d6dd7ab9367c9abf357856d7158fd8a1f5830936b8f327d7a"}
Nov 03 22:58:22.564260 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:22.564190    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-15e7a7-79b4b8dbd4-pb92m" event={"ID":"cfb72051-95d9-406f-b200-e676af52c8ab","Type":"ContainerStarted","Data":"2bdd7e4efa77c8c47bf35029d24c7b24216710e5ee2fa20a0ebe4a59aa96e640"}
Nov 03 22:58:22.564260 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:22.564227    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-15e7a7-79b4b8dbd4-pb92m"
Nov 03 22:58:22.603548 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:22.603486    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15e7a7-5467566b7-npkkl" podStartSLOduration=1.60346737 podStartE2EDuration="1.60346737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:58:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:58:22.601565397 +0000 UTC m=+2726.627421261" watchObservedRunningTime="2026-11-03 22:58:22.60346737 +0000 UTC m=+2726.629323234"
Nov 03 22:58:22.643736 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:22.643684    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-15e7a7-79b4b8dbd4-pb92m" podStartSLOduration=1.643670135 podStartE2EDuration="1.643670135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:58:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:58:22.640572774 +0000 UTC m=+2726.666428635" watchObservedRunningTime="2026-11-03 22:58:22.643670135 +0000 UTC m=+2726.669526000"
Nov 03 22:58:23.181851 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:23.181816    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:58:23.749000 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:23.748961    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:58:23.894010 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:23.893975    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:58:24.822083 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:24.822040    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:58:26.408054 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:26.407995    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:58:28.414412 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:28.414377    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:58:30.026443 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:30.026407    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:58:31.260763 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:58:31.260728    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:58:31.609534 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:31.609450    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:58:33.074679 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:33.074642    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:58:33.574461 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:33.574432    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15e7a7-5467566b7-npkkl"
Nov 03 22:58:33.576274 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:33.576244    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-15e7a7-79b4b8dbd4-pb92m"
Nov 03 22:58:34.948149 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:34.948093    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:58:36.725047 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:36.725013    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:58:38.366431 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:38.366392    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:58:38.504679 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:38.504648    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:58:39.999942 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:39.999904    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:58:41.708249 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:41.708188    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:58:43.356052 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:43.356004    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:58:45.019068 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:45.019037    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:58:45.763812 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:45.763782    4431 generic.go:356] "Generic (PLEG): container finished" podID="681f7328-e1f5-4544-876d-410f30ee83e5" containerID="56cd9f39de4e5a13cba87a2ef1dcb98947e79cdbcb8bc3c4e4731bb1c4248148" exitCode=0
Nov 03 22:58:45.763995 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:45.763860    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2" event={"ID":"681f7328-e1f5-4544-876d-410f30ee83e5","Type":"ContainerDied","Data":"56cd9f39de4e5a13cba87a2ef1dcb98947e79cdbcb8bc3c4e4731bb1c4248148"}
Nov 03 22:58:45.764453 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:45.764433    4431 scope.go:122] "RemoveContainer" containerID="56cd9f39de4e5a13cba87a2ef1dcb98947e79cdbcb8bc3c4e4731bb1c4248148"
Nov 03 22:58:45.766360 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:45.766330    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:58:45.766490 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:45.766415    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:58:45.766490 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:45.766465    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:58:46.102610 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:46.102529    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2"
Nov 03 22:58:46.697870 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:46.697836    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:58:46.774224 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:46.774175    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2" event={"ID":"681f7328-e1f5-4544-876d-410f30ee83e5","Type":"ContainerStarted","Data":"eb013844454b316fbcb6950f42574565ea2b70fa136c40a95058fb2bb79aeb47"}
Nov 03 22:58:47.595443 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:47.595410    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2"
Nov 03 22:58:47.597097 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:47.597070    4431 patch_prober.go:29] interesting pod/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Nov 03 22:58:47.597244 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:47.597125    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2" podUID="681f7328-e1f5-4544-876d-410f30ee83e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Nov 03 22:58:48.230333 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:48.230297    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:58:48.595965 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:48.595881    4431 patch_prober.go:29] interesting pod/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Nov 03 22:58:48.595965 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:48.595941    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2" podUID="681f7328-e1f5-4544-876d-410f30ee83e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Nov 03 22:58:49.596665 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:49.596627    4431 patch_prober.go:29] interesting pod/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Nov 03 22:58:49.597314 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:49.597277    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2" podUID="681f7328-e1f5-4544-876d-410f30ee83e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Nov 03 22:58:49.868367 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:49.868279    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:58:50.010237 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:50.009792    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:58:50.595288 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:50.595249    4431 patch_prober.go:29] interesting pod/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Nov 03 22:58:50.595462 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:50.595332    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2" podUID="681f7328-e1f5-4544-876d-410f30ee83e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Nov 03 22:58:51.595441 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:51.595407    4431 patch_prober.go:29] interesting pod/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Nov 03 22:58:51.595861 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:51.595459    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2" podUID="681f7328-e1f5-4544-876d-410f30ee83e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Nov 03 22:58:51.698009 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:51.697968    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:58:52.600755 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:52.600723    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2"
Nov 03 22:58:52.601183 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:52.601141    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2"
Nov 03 22:58:52.601919 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:52.601897    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2"
Nov 03 22:58:53.084422 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:53.084391    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ff80c0-6794988c8d-w644b"]
Nov 03 22:58:53.254340 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:53.254296    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:58:53.399368 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:53.399278    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:58:53.544932 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:53.544894    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:58:54.889428 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:54.889390    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:58:55.029594 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:55.029560    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:58:56.404529 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:56.404499    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:58:57.865975 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:57.865940    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e7a7-5fc59b578-928wg"]
Nov 03 22:58:57.870445 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:57.870427    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e7a7-5fc59b578-928wg"
Nov 03 22:58:57.890086 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:57.890055    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-15e7a7-serving-cert\""
Nov 03 22:58:57.894750 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:57.894720    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e7a7-5fc59b578-928wg"]
Nov 03 22:58:57.983126 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:57.983083    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:58:57.992526 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:57.992499    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xgtr\" (UniqueName: \"kubernetes.io/projected/9684a170-ee87-4c62-943f-86816632125e-kube-api-access-6xgtr\") pod \"maas-api-e2e-ipp-a-15e7a7-5fc59b578-928wg\" (UID: \"9684a170-ee87-4c62-943f-86816632125e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e7a7-5fc59b578-928wg"
Nov 03 22:58:57.992640 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:57.992556    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9684a170-ee87-4c62-943f-86816632125e-metrics-certs\") pod \"maas-api-e2e-ipp-a-15e7a7-5fc59b578-928wg\" (UID: \"9684a170-ee87-4c62-943f-86816632125e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e7a7-5fc59b578-928wg"
Nov 03 22:58:57.992682 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:57.992639    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9684a170-ee87-4c62-943f-86816632125e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-15e7a7-5fc59b578-928wg\" (UID: \"9684a170-ee87-4c62-943f-86816632125e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e7a7-5fc59b578-928wg"
Nov 03 22:58:57.992777 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:57.992761    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9684a170-ee87-4c62-943f-86816632125e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-15e7a7-5fc59b578-928wg\" (UID: \"9684a170-ee87-4c62-943f-86816632125e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e7a7-5fc59b578-928wg"
Nov 03 22:58:58.085540 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:58.085501    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-ff80c0-6794988c8d-w644b" podUID="5ce45ae5-a3a7-4df1-8052-30c8f0512cf6" containerName="payload-processing" containerID="cri-o://518b6ebd709fa891c25dbd028c034cdcd42b90977e4894b00cc33e601721eecd" gracePeriod=25
Nov 03 22:58:58.094503 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:58.094472    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xgtr\" (UniqueName: \"kubernetes.io/projected/9684a170-ee87-4c62-943f-86816632125e-kube-api-access-6xgtr\") pod \"maas-api-e2e-ipp-a-15e7a7-5fc59b578-928wg\" (UID: \"9684a170-ee87-4c62-943f-86816632125e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e7a7-5fc59b578-928wg"
Nov 03 22:58:58.094636 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:58.094551    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9684a170-ee87-4c62-943f-86816632125e-metrics-certs\") pod \"maas-api-e2e-ipp-a-15e7a7-5fc59b578-928wg\" (UID: \"9684a170-ee87-4c62-943f-86816632125e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e7a7-5fc59b578-928wg"
Nov 03 22:58:58.094636 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:58.094587    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9684a170-ee87-4c62-943f-86816632125e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-15e7a7-5fc59b578-928wg\" (UID: \"9684a170-ee87-4c62-943f-86816632125e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e7a7-5fc59b578-928wg"
Nov 03 22:58:58.094758 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:58.094698    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9684a170-ee87-4c62-943f-86816632125e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-15e7a7-5fc59b578-928wg\" (UID: \"9684a170-ee87-4c62-943f-86816632125e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e7a7-5fc59b578-928wg"
Nov 03 22:58:58.095330 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:58.095304    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9684a170-ee87-4c62-943f-86816632125e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-15e7a7-5fc59b578-928wg\" (UID: \"9684a170-ee87-4c62-943f-86816632125e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e7a7-5fc59b578-928wg"
Nov 03 22:58:58.096952 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:58.096924    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9684a170-ee87-4c62-943f-86816632125e-metrics-certs\") pod \"maas-api-e2e-ipp-a-15e7a7-5fc59b578-928wg\" (UID: \"9684a170-ee87-4c62-943f-86816632125e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e7a7-5fc59b578-928wg"
Nov 03 22:58:58.097130 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:58.097106    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9684a170-ee87-4c62-943f-86816632125e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-15e7a7-5fc59b578-928wg\" (UID: \"9684a170-ee87-4c62-943f-86816632125e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e7a7-5fc59b578-928wg"
Nov 03 22:58:58.117405 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:58.117346    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xgtr\" (UniqueName: \"kubernetes.io/projected/9684a170-ee87-4c62-943f-86816632125e-kube-api-access-6xgtr\") pod \"maas-api-e2e-ipp-a-15e7a7-5fc59b578-928wg\" (UID: \"9684a170-ee87-4c62-943f-86816632125e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e7a7-5fc59b578-928wg"
Nov 03 22:58:58.122945 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:58.122920    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:58:58.185488 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:58.185456    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e7a7-5fc59b578-928wg"
Nov 03 22:58:58.402151 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:58.402125    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ff80c0-6794988c8d-w644b"
Nov 03 22:58:58.500163 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:58.500129    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ce45ae5-a3a7-4df1-8052-30c8f0512cf6-kube-api-access-p4k5p\" (UniqueName: \"kubernetes.io/projected/5ce45ae5-a3a7-4df1-8052-30c8f0512cf6-kube-api-access-p4k5p\") pod \"5ce45ae5-a3a7-4df1-8052-30c8f0512cf6\" (UID: \"5ce45ae5-a3a7-4df1-8052-30c8f0512cf6\") "
Nov 03 22:58:58.500470 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:58.500365    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ce45ae5-a3a7-4df1-8052-30c8f0512cf6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ce45ae5-a3a7-4df1-8052-30c8f0512cf6-plugins-config-volume\") pod \"5ce45ae5-a3a7-4df1-8052-30c8f0512cf6\" (UID: \"5ce45ae5-a3a7-4df1-8052-30c8f0512cf6\") "
Nov 03 22:58:58.500776 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:58.500752    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ce45ae5-a3a7-4df1-8052-30c8f0512cf6-plugins-config-volume" pod "5ce45ae5-a3a7-4df1-8052-30c8f0512cf6" (UID: "5ce45ae5-a3a7-4df1-8052-30c8f0512cf6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:58:58.501006 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:58.500988    4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ce45ae5-a3a7-4df1-8052-30c8f0512cf6-plugins-config-volume\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:58:58.502260 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:58.502233    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ce45ae5-a3a7-4df1-8052-30c8f0512cf6-kube-api-access-p4k5p" pod "5ce45ae5-a3a7-4df1-8052-30c8f0512cf6" (UID: "5ce45ae5-a3a7-4df1-8052-30c8f0512cf6"). InnerVolumeSpecName "kube-api-access-p4k5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:58:58.502697 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:58.502677    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e7a7-5fc59b578-928wg"]
Nov 03 22:58:58.503548 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:58:58.503525    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9684a170_ee87_4c62_943f_86816632125e.slice/crio-531d3cf2b3fdef1e685afb73259462477244d1af9cd8d22428adb37c6672d069 WatchSource:0}: Error finding container 531d3cf2b3fdef1e685afb73259462477244d1af9cd8d22428adb37c6672d069: Status 404 returned error can't find the container with id 531d3cf2b3fdef1e685afb73259462477244d1af9cd8d22428adb37c6672d069
Nov 03 22:58:58.505017 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:58.505000    4431 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 22:58:58.602627 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:58.602597    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4k5p\" (UniqueName: \"kubernetes.io/projected/5ce45ae5-a3a7-4df1-8052-30c8f0512cf6-kube-api-access-p4k5p\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:58:58.860523 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:58.860477    4431 generic.go:356] "Generic (PLEG): container finished" podID="5ce45ae5-a3a7-4df1-8052-30c8f0512cf6" containerID="518b6ebd709fa891c25dbd028c034cdcd42b90977e4894b00cc33e601721eecd" exitCode=0
Nov 03 22:58:58.860719 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:58.860579    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ff80c0-6794988c8d-w644b"
Nov 03 22:58:58.860719 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:58.860581    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ff80c0-6794988c8d-w644b" event={"ID":"5ce45ae5-a3a7-4df1-8052-30c8f0512cf6","Type":"ContainerDied","Data":"518b6ebd709fa891c25dbd028c034cdcd42b90977e4894b00cc33e601721eecd"}
Nov 03 22:58:58.860719 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:58.860712    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ff80c0-6794988c8d-w644b" event={"ID":"5ce45ae5-a3a7-4df1-8052-30c8f0512cf6","Type":"ContainerDied","Data":"433508f2d9d8fc4f1428911ff495bbc30e12dff59f759b7b05fdc9d75e63046f"}
Nov 03 22:58:58.860888 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:58.860743    4431 scope.go:122] "RemoveContainer" containerID="518b6ebd709fa891c25dbd028c034cdcd42b90977e4894b00cc33e601721eecd"
Nov 03 22:58:58.862189 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:58.862168    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e7a7-5fc59b578-928wg" event={"ID":"9684a170-ee87-4c62-943f-86816632125e","Type":"ContainerStarted","Data":"531d3cf2b3fdef1e685afb73259462477244d1af9cd8d22428adb37c6672d069"}
Nov 03 22:58:58.873305 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:58.873288    4431 scope.go:122] "RemoveContainer" containerID="518b6ebd709fa891c25dbd028c034cdcd42b90977e4894b00cc33e601721eecd"
Nov 03 22:58:58.873868 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:58:58.873843    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"518b6ebd709fa891c25dbd028c034cdcd42b90977e4894b00cc33e601721eecd\": container with ID starting with 518b6ebd709fa891c25dbd028c034cdcd42b90977e4894b00cc33e601721eecd not found: ID does not exist" containerID="518b6ebd709fa891c25dbd028c034cdcd42b90977e4894b00cc33e601721eecd"
Nov 03 22:58:58.873950 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:58.873875    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"518b6ebd709fa891c25dbd028c034cdcd42b90977e4894b00cc33e601721eecd"} err="failed to get container status \"518b6ebd709fa891c25dbd028c034cdcd42b90977e4894b00cc33e601721eecd\": rpc error: code = NotFound desc = could not find container \"518b6ebd709fa891c25dbd028c034cdcd42b90977e4894b00cc33e601721eecd\": container with ID starting with 518b6ebd709fa891c25dbd028c034cdcd42b90977e4894b00cc33e601721eecd not found: ID does not exist"
Nov 03 22:58:58.913902 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:58.913877    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ff80c0-6794988c8d-w644b"]
Nov 03 22:58:58.925163 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:58.925135    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ff80c0-6794988c8d-w644b"]
Nov 03 22:58:59.711189 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:59.711153    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:58:59.871297 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:59.871252    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e7a7-5fc59b578-928wg" event={"ID":"9684a170-ee87-4c62-943f-86816632125e","Type":"ContainerStarted","Data":"4296695b57061563991dd274ee13b832680949eddc5a9b9569f6935b55ada396"}
Nov 03 22:58:59.871487 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:59.871324    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e7a7-5fc59b578-928wg"
Nov 03 22:58:59.915011 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:58:59.914958    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e7a7-5fc59b578-928wg" podStartSLOduration=2.51289295 podStartE2EDuration="2.914944492s" totalImagesPullingTime="402.051542ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:58:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:58:59.913720602 +0000 UTC m=+2763.939576464" watchObservedRunningTime="2026-11-03 22:58:59.914944492 +0000 UTC m=+2763.940800400"
Nov 03 22:59:00.120894 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:00.120810    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ce45ae5-a3a7-4df1-8052-30c8f0512cf6" path="/var/lib/kubelet/pods/5ce45ae5-a3a7-4df1-8052-30c8f0512cf6/volumes"
Nov 03 22:59:01.241802 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:01.241772    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:59:01.266297 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:59:01.266257    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:59:03.221411 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:03.221357    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:59:04.708804 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:04.708768    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:59:05.883579 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:05.883551    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e7a7-5fc59b578-928wg"
Nov 03 22:59:06.398827 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:06.398796    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:59:08.062030 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:08.061995    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:59:10.208539 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:10.208505    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:59:10.714689 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:10.714640    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7266-37389c"]
Nov 03 22:59:10.715348 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:10.715332    4431 cpu_manager.go:414] "containerMap: removing container" podUID="5ce45ae5-a3a7-4df1-8052-30c8f0512cf6" containerName="payload-processing"
Nov 03 22:59:10.715400 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:10.715348    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ce45ae5-a3a7-4df1-8052-30c8f0512cf6" containerName="payload-processing"
Nov 03 22:59:10.715485 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:10.715475    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ce45ae5-a3a7-4df1-8052-30c8f0512cf6" containerName="payload-processing"
Nov 03 22:59:10.720264 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:10.720210    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7266-37389c"
Nov 03 22:59:10.734532 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:10.734506    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7266-37389c"]
Nov 03 22:59:10.742252 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:10.742207    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-khmwt\""
Nov 03 22:59:10.887789 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:10.887750    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtvbs\" (UniqueName: \"kubernetes.io/projected/0b19d9f9-17b7-4140-a82c-4942facea4d2-kube-api-access-vtvbs\") pod \"test-curl-7266-37389c\" (UID: \"0b19d9f9-17b7-4140-a82c-4942facea4d2\") " pod="opendatahub/test-curl-7266-37389c"
Nov 03 22:59:10.989670 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:10.989574    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtvbs\" (UniqueName: \"kubernetes.io/projected/0b19d9f9-17b7-4140-a82c-4942facea4d2-kube-api-access-vtvbs\") pod \"test-curl-7266-37389c\" (UID: \"0b19d9f9-17b7-4140-a82c-4942facea4d2\") " pod="opendatahub/test-curl-7266-37389c"
Nov 03 22:59:11.012650 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:11.012624    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtvbs\" (UniqueName: \"kubernetes.io/projected/0b19d9f9-17b7-4140-a82c-4942facea4d2-kube-api-access-vtvbs\") pod \"test-curl-7266-37389c\" (UID: \"0b19d9f9-17b7-4140-a82c-4942facea4d2\") " pod="opendatahub/test-curl-7266-37389c"
Nov 03 22:59:11.032928 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:11.032903    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7266-37389c"
Nov 03 22:59:11.337535 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:11.337477    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7266-37389c"]
Nov 03 22:59:11.340339 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:59:11.340307    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0b19d9f9_17b7_4140_a82c_4942facea4d2.slice/crio-7bc0d06b023012af648c85104326779f0548e5e920002a56f279a7c942847d3e WatchSource:0}: Error finding container 7bc0d06b023012af648c85104326779f0548e5e920002a56f279a7c942847d3e: Status 404 returned error can't find the container with id 7bc0d06b023012af648c85104326779f0548e5e920002a56f279a7c942847d3e
Nov 03 22:59:11.878716 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:11.878685    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:59:11.953173 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:11.953128    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7266-37389c" event={"ID":"0b19d9f9-17b7-4140-a82c-4942facea4d2","Type":"ContainerStarted","Data":"213a49b1c741ca485ba855b43b8acbedbe849b7d5f4d1474ee47b00e13b5ea05"}
Nov 03 22:59:11.953173 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:11.953179    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7266-37389c" event={"ID":"0b19d9f9-17b7-4140-a82c-4942facea4d2","Type":"ContainerStarted","Data":"7bc0d06b023012af648c85104326779f0548e5e920002a56f279a7c942847d3e"}
Nov 03 22:59:11.992889 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:11.992826    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7266-37389c" podStartSLOduration=1.5147014840000002 podStartE2EDuration="1.992810184s" totalImagesPullingTime="478.1087ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:59:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:59:11.990943999 +0000 UTC m=+2776.016799861" watchObservedRunningTime="2026-11-03 22:59:11.992810184 +0000 UTC m=+2776.018666046"
Nov 03 22:59:12.862705 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:12.862655    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7266-37389c"]
Nov 03 22:59:12.867679 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:12.867648    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7266-37389c"]
Nov 03 22:59:13.660900 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:13.660865    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:59:13.966986 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:13.966929    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7266-37389c" podUID="0b19d9f9-17b7-4140-a82c-4942facea4d2" containerName="test-curl-7266-37389c" containerID="cri-o://213a49b1c741ca485ba855b43b8acbedbe849b7d5f4d1474ee47b00e13b5ea05" gracePeriod=2
Nov 03 22:59:13.980413 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:59:13.980377    4431 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7266-37389c\" is forbidden: User \"system:node:ip-10-0-139-73.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-73.ec2.internal' and this object" podUID="0b19d9f9-17b7-4140-a82c-4942facea4d2" pod="opendatahub/test-curl-7266-37389c"
Nov 03 22:59:15.237357 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:15.237316    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:59:16.196030 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:16.195998    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7266-37389c"
Nov 03 22:59:16.209690 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:59:16.209656    4431 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7266-37389c\" is forbidden: User \"system:node:ip-10-0-139-73.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-73.ec2.internal' and this object" podUID="0b19d9f9-17b7-4140-a82c-4942facea4d2" pod="opendatahub/test-curl-7266-37389c"
Nov 03 22:59:16.265475 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:16.265402    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b19d9f9-17b7-4140-a82c-4942facea4d2-kube-api-access-vtvbs\" (UniqueName: \"kubernetes.io/projected/0b19d9f9-17b7-4140-a82c-4942facea4d2-kube-api-access-vtvbs\") pod \"0b19d9f9-17b7-4140-a82c-4942facea4d2\" (UID: \"0b19d9f9-17b7-4140-a82c-4942facea4d2\") "
Nov 03 22:59:16.267630 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:16.267597    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b19d9f9-17b7-4140-a82c-4942facea4d2-kube-api-access-vtvbs" pod "0b19d9f9-17b7-4140-a82c-4942facea4d2" (UID: "0b19d9f9-17b7-4140-a82c-4942facea4d2"). InnerVolumeSpecName "kube-api-access-vtvbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:59:16.367581 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:16.367547    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtvbs\" (UniqueName: \"kubernetes.io/projected/0b19d9f9-17b7-4140-a82c-4942facea4d2-kube-api-access-vtvbs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:59:16.901830 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:16.901798    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:59:16.987663 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:16.987627    4431 generic.go:356] "Generic (PLEG): container finished" podID="0b19d9f9-17b7-4140-a82c-4942facea4d2" containerID="213a49b1c741ca485ba855b43b8acbedbe849b7d5f4d1474ee47b00e13b5ea05" exitCode=137
Nov 03 22:59:16.987846 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:16.987668    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7266-37389c"
Nov 03 22:59:16.987846 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:16.987715    4431 scope.go:122] "RemoveContainer" containerID="213a49b1c741ca485ba855b43b8acbedbe849b7d5f4d1474ee47b00e13b5ea05"
Nov 03 22:59:16.997751 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:16.997737    4431 scope.go:122] "RemoveContainer" containerID="213a49b1c741ca485ba855b43b8acbedbe849b7d5f4d1474ee47b00e13b5ea05"
Nov 03 22:59:16.998008 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:59:16.997985    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"213a49b1c741ca485ba855b43b8acbedbe849b7d5f4d1474ee47b00e13b5ea05\": container with ID starting with 213a49b1c741ca485ba855b43b8acbedbe849b7d5f4d1474ee47b00e13b5ea05 not found: ID does not exist" containerID="213a49b1c741ca485ba855b43b8acbedbe849b7d5f4d1474ee47b00e13b5ea05"
Nov 03 22:59:16.998089 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:16.998016    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"213a49b1c741ca485ba855b43b8acbedbe849b7d5f4d1474ee47b00e13b5ea05"} err="failed to get container status \"213a49b1c741ca485ba855b43b8acbedbe849b7d5f4d1474ee47b00e13b5ea05\": rpc error: code = NotFound desc = could not find container \"213a49b1c741ca485ba855b43b8acbedbe849b7d5f4d1474ee47b00e13b5ea05\": container with ID starting with 213a49b1c741ca485ba855b43b8acbedbe849b7d5f4d1474ee47b00e13b5ea05 not found: ID does not exist"
Nov 03 22:59:17.001305 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:59:17.001272    4431 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7266-37389c\" is forbidden: User \"system:node:ip-10-0-139-73.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-73.ec2.internal' and this object" podUID="0b19d9f9-17b7-4140-a82c-4942facea4d2" pod="opendatahub/test-curl-7266-37389c"
Nov 03 22:59:17.014435 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:59:17.014407    4431 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7266-37389c\" is forbidden: User \"system:node:ip-10-0-139-73.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-73.ec2.internal' and this object" podUID="0b19d9f9-17b7-4140-a82c-4942facea4d2" pod="opendatahub/test-curl-7266-37389c"
Nov 03 22:59:18.088540 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:18.088510    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b19d9f9-17b7-4140-a82c-4942facea4d2" path="/var/lib/kubelet/pods/0b19d9f9-17b7-4140-a82c-4942facea4d2/volumes"
Nov 03 22:59:18.491066 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:18.491028    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:59:20.014384 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:20.014331    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:59:21.715174 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:21.715132    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:59:22.831895 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:22.831864    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:59:23.417078 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:23.417042    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:59:23.930151 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:23.930114    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:59:25.003863 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:25.003824    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:59:26.604162 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:26.604126    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:59:26.880277 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:26.880188    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f48f46-7c74fffd4b-b86lk"]
Nov 03 22:59:26.910879 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:26.910836    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f48f46-9875d54dc-m8hbh"]
Nov 03 22:59:27.068779 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:27.068725    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:59:27.221282 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:27.221245    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:59:28.525661 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:28.525631    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:59:30.121410 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:30.121374    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:59:31.182393 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:31.182357    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:59:31.266158 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:59:31.266118    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 22:59:31.324787 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:31.324749    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:59:31.881462 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:31.881414    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-f48f46-7c74fffd4b-b86lk" podUID="3fe7c171-abe2-428d-81e1-d13da5f0c376" containerName="payload-processing" containerID="cri-o://9a0337b91aaa74144c37601fccdb09507ade8eba9247b7a2b1d1db9a3aba9541" gracePeriod=25
Nov 03 22:59:31.911142 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:31.911106    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f48f46-9875d54dc-m8hbh" podUID="605d2d22-11a8-420e-9776-f4d173819a04" containerName="payload-pre-processing" containerID="cri-o://9915d1ed9660ace2cda9e402987737878e704963d2e535703226bc5376b0f07d" gracePeriod=25
Nov 03 22:59:32.115107 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:32.115072    4431 generic.go:356] "Generic (PLEG): container finished" podID="605d2d22-11a8-420e-9776-f4d173819a04" containerID="9915d1ed9660ace2cda9e402987737878e704963d2e535703226bc5376b0f07d" exitCode=0
Nov 03 22:59:32.115338 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:32.115232    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f48f46-9875d54dc-m8hbh" event={"ID":"605d2d22-11a8-420e-9776-f4d173819a04","Type":"ContainerDied","Data":"9915d1ed9660ace2cda9e402987737878e704963d2e535703226bc5376b0f07d"}
Nov 03 22:59:32.117434 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:32.117406    4431 generic.go:356] "Generic (PLEG): container finished" podID="3fe7c171-abe2-428d-81e1-d13da5f0c376" containerID="9a0337b91aaa74144c37601fccdb09507ade8eba9247b7a2b1d1db9a3aba9541" exitCode=0
Nov 03 22:59:32.117660 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:32.117644    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f48f46-7c74fffd4b-b86lk" event={"ID":"3fe7c171-abe2-428d-81e1-d13da5f0c376","Type":"ContainerDied","Data":"9a0337b91aaa74144c37601fccdb09507ade8eba9247b7a2b1d1db9a3aba9541"}
Nov 03 22:59:32.211632 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:32.211595    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:59:32.212097 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:32.211923    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f48f46-7c74fffd4b-b86lk"
Nov 03 22:59:32.256947 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:32.256904    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f48f46-9875d54dc-m8hbh"
Nov 03 22:59:32.303682 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:32.303652    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fe7c171-abe2-428d-81e1-d13da5f0c376-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fe7c171-abe2-428d-81e1-d13da5f0c376-plugins-config-volume\") pod \"3fe7c171-abe2-428d-81e1-d13da5f0c376\" (UID: \"3fe7c171-abe2-428d-81e1-d13da5f0c376\") "
Nov 03 22:59:32.303890 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:32.303809    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fe7c171-abe2-428d-81e1-d13da5f0c376-kube-api-access-vzc7c\" (UniqueName: \"kubernetes.io/projected/3fe7c171-abe2-428d-81e1-d13da5f0c376-kube-api-access-vzc7c\") pod \"3fe7c171-abe2-428d-81e1-d13da5f0c376\" (UID: \"3fe7c171-abe2-428d-81e1-d13da5f0c376\") "
Nov 03 22:59:32.304121 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:32.304095    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fe7c171-abe2-428d-81e1-d13da5f0c376-plugins-config-volume" pod "3fe7c171-abe2-428d-81e1-d13da5f0c376" (UID: "3fe7c171-abe2-428d-81e1-d13da5f0c376"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:59:32.304459 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:32.304437    4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fe7c171-abe2-428d-81e1-d13da5f0c376-plugins-config-volume\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:59:32.306492 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:32.306458    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fe7c171-abe2-428d-81e1-d13da5f0c376-kube-api-access-vzc7c" pod "3fe7c171-abe2-428d-81e1-d13da5f0c376" (UID: "3fe7c171-abe2-428d-81e1-d13da5f0c376"). InnerVolumeSpecName "kube-api-access-vzc7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:59:32.405661 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:32.405628    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/605d2d22-11a8-420e-9776-f4d173819a04-kube-api-access-dgwlt\" (UniqueName: \"kubernetes.io/projected/605d2d22-11a8-420e-9776-f4d173819a04-kube-api-access-dgwlt\") pod \"605d2d22-11a8-420e-9776-f4d173819a04\" (UID: \"605d2d22-11a8-420e-9776-f4d173819a04\") "
Nov 03 22:59:32.405828 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:32.405674    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/605d2d22-11a8-420e-9776-f4d173819a04-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/605d2d22-11a8-420e-9776-f4d173819a04-plugins-config-volume\") pod \"605d2d22-11a8-420e-9776-f4d173819a04\" (UID: \"605d2d22-11a8-420e-9776-f4d173819a04\") "
Nov 03 22:59:32.406063 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:32.406043    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/605d2d22-11a8-420e-9776-f4d173819a04-plugins-config-volume" pod "605d2d22-11a8-420e-9776-f4d173819a04" (UID: "605d2d22-11a8-420e-9776-f4d173819a04"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:59:32.406276 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:32.406262    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzc7c\" (UniqueName: \"kubernetes.io/projected/3fe7c171-abe2-428d-81e1-d13da5f0c376-kube-api-access-vzc7c\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:59:32.406331 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:32.406278    4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/605d2d22-11a8-420e-9776-f4d173819a04-plugins-config-volume\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:59:32.408424 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:32.408390    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/605d2d22-11a8-420e-9776-f4d173819a04-kube-api-access-dgwlt" pod "605d2d22-11a8-420e-9776-f4d173819a04" (UID: "605d2d22-11a8-420e-9776-f4d173819a04"). InnerVolumeSpecName "kube-api-access-dgwlt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:59:32.507907 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:32.507830    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgwlt\" (UniqueName: \"kubernetes.io/projected/605d2d22-11a8-420e-9776-f4d173819a04-kube-api-access-dgwlt\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:59:33.133237 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:33.129492    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f48f46-9875d54dc-m8hbh" event={"ID":"605d2d22-11a8-420e-9776-f4d173819a04","Type":"ContainerDied","Data":"4cded80e934f3dd56412f910c8727731c6d7e11a279cdcc6c6ae811595ff4f0d"}
Nov 03 22:59:33.133237 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:33.129493    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f48f46-9875d54dc-m8hbh"
Nov 03 22:59:33.133237 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:33.129563    4431 scope.go:122] "RemoveContainer" containerID="9915d1ed9660ace2cda9e402987737878e704963d2e535703226bc5376b0f07d"
Nov 03 22:59:33.133237 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:33.131345    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f48f46-7c74fffd4b-b86lk" event={"ID":"3fe7c171-abe2-428d-81e1-d13da5f0c376","Type":"ContainerDied","Data":"b168fbaf18bc09ebf6d222d7b556413d40ad9cffd85aff74b6312a0451b4d401"}
Nov 03 22:59:33.133237 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:33.131434    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f48f46-7c74fffd4b-b86lk"
Nov 03 22:59:33.152202 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:33.152177    4431 scope.go:122] "RemoveContainer" containerID="9a0337b91aaa74144c37601fccdb09507ade8eba9247b7a2b1d1db9a3aba9541"
Nov 03 22:59:33.183293 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:33.183266    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f48f46-9875d54dc-m8hbh"]
Nov 03 22:59:33.195247 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:33.195197    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f48f46-9875d54dc-m8hbh"]
Nov 03 22:59:33.228910 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:33.228873    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f48f46-7c74fffd4b-b86lk"]
Nov 03 22:59:33.233378 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:33.233352    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f48f46-7c74fffd4b-b86lk"]
Nov 03 22:59:33.866438 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:33.866403    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:59:34.088924 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:34.088892    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fe7c171-abe2-428d-81e1-d13da5f0c376" path="/var/lib/kubelet/pods/3fe7c171-abe2-428d-81e1-d13da5f0c376/volumes"
Nov 03 22:59:34.089361 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:34.089348    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="605d2d22-11a8-420e-9776-f4d173819a04" path="/var/lib/kubelet/pods/605d2d22-11a8-420e-9776-f4d173819a04/volumes"
Nov 03 22:59:35.432490 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:35.432450    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:59:36.588462 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:36.588417    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:59:37.116148 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:37.116112    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:59:38.596147 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:38.596082    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:59:40.206631 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:40.206598    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:59:41.846082 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:41.846046    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:59:43.530522 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:43.530390    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:59:43.713137 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:43.713093    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:59:43.973423 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:43.973388    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-b023a2/auth-test-model-b023a2-kserve-7f85457dc7-pvx2r"]
Nov 03 22:59:44.040421 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:44.040372    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:59:44.855883 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:44.855845    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:59:45.763081 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:45.763045    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:59:45.907007 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:45.906975    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:59:46.275451 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:46.275402    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:59:46.371748 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:46.371701    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-15e7a7-79b4b8dbd4-pb92m"]
Nov 03 22:59:46.418224 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:46.418192    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-15e7a7-5467566b7-npkkl"]
Nov 03 22:59:46.545913 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:46.545834    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:59:46.688328 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:46.688282    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:59:47.613710 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:47.613670    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:59:49.488051 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:49.488005    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:59:49.982561 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:49.982523    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:59:50.130773 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:50.130735    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:59:51.263545 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:51.263512    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:59:51.372412 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:51.372358    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-15e7a7-79b4b8dbd4-pb92m" podUID="cfb72051-95d9-406f-b200-e676af52c8ab" containerName="payload-processing" containerID="cri-o://08a2498654efb31d6dd7ab9367c9abf357856d7158fd8a1f5830936b8f327d7a" gracePeriod=25
Nov 03 22:59:51.419620 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:51.419587    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15e7a7-5467566b7-npkkl" podUID="e2a90093-86e4-4bed-9892-31150b35ff48" containerName="payload-pre-processing" containerID="cri-o://0dc27536e80b98489f5e1dafd8154c7feae26013a8b445c2af60a2cf9fcc657d" gracePeriod=25
Nov 03 22:59:51.714748 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:51.714704    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-15e7a7-79b4b8dbd4-pb92m"
Nov 03 22:59:51.742752 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:51.742690    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:59:51.757678 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:51.757655    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15e7a7-5467566b7-npkkl"
Nov 03 22:59:51.819556 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:51.819465    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfb72051-95d9-406f-b200-e676af52c8ab-kube-api-access-pxnkt\" (UniqueName: \"kubernetes.io/projected/cfb72051-95d9-406f-b200-e676af52c8ab-kube-api-access-pxnkt\") pod \"cfb72051-95d9-406f-b200-e676af52c8ab\" (UID: \"cfb72051-95d9-406f-b200-e676af52c8ab\") "
Nov 03 22:59:51.819732 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:51.819719    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfb72051-95d9-406f-b200-e676af52c8ab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfb72051-95d9-406f-b200-e676af52c8ab-plugins-config-volume\") pod \"cfb72051-95d9-406f-b200-e676af52c8ab\" (UID: \"cfb72051-95d9-406f-b200-e676af52c8ab\") "
Nov 03 22:59:51.820137 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:51.820108    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfb72051-95d9-406f-b200-e676af52c8ab-plugins-config-volume" pod "cfb72051-95d9-406f-b200-e676af52c8ab" (UID: "cfb72051-95d9-406f-b200-e676af52c8ab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:59:51.821847 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:51.821821    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfb72051-95d9-406f-b200-e676af52c8ab-kube-api-access-pxnkt" pod "cfb72051-95d9-406f-b200-e676af52c8ab" (UID: "cfb72051-95d9-406f-b200-e676af52c8ab"). InnerVolumeSpecName "kube-api-access-pxnkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:59:51.838735 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:51.838700    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4a01f9-57fb584d-fn5pv"]
Nov 03 22:59:51.839707 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:51.839679    4431 cpu_manager.go:414] "containerMap: removing container" podUID="3fe7c171-abe2-428d-81e1-d13da5f0c376" containerName="payload-processing"
Nov 03 22:59:51.839707 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:51.839708    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fe7c171-abe2-428d-81e1-d13da5f0c376" containerName="payload-processing"
Nov 03 22:59:51.839854 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:51.839772    4431 cpu_manager.go:414] "containerMap: removing container" podUID="e2a90093-86e4-4bed-9892-31150b35ff48" containerName="payload-pre-processing"
Nov 03 22:59:51.839854 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:51.839781    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2a90093-86e4-4bed-9892-31150b35ff48" containerName="payload-pre-processing"
Nov 03 22:59:51.839854 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:51.839813    4431 cpu_manager.go:414] "containerMap: removing container" podUID="605d2d22-11a8-420e-9776-f4d173819a04" containerName="payload-pre-processing"
Nov 03 22:59:51.839854 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:51.839821    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="605d2d22-11a8-420e-9776-f4d173819a04" containerName="payload-pre-processing"
Nov 03 22:59:51.839854 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:51.839839    4431 cpu_manager.go:414] "containerMap: removing container" podUID="cfb72051-95d9-406f-b200-e676af52c8ab" containerName="payload-processing"
Nov 03 22:59:51.839854 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:51.839847    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfb72051-95d9-406f-b200-e676af52c8ab" containerName="payload-processing"
Nov 03 22:59:51.840160 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:51.839865    4431 cpu_manager.go:414] "containerMap: removing container" podUID="0b19d9f9-17b7-4140-a82c-4942facea4d2" containerName="test-curl-7266-37389c"
Nov 03 22:59:51.840160 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:51.839873    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b19d9f9-17b7-4140-a82c-4942facea4d2" containerName="test-curl-7266-37389c"
Nov 03 22:59:51.840160 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:51.840052    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="605d2d22-11a8-420e-9776-f4d173819a04" containerName="payload-pre-processing"
Nov 03 22:59:51.840160 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:51.840069    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfb72051-95d9-406f-b200-e676af52c8ab" containerName="payload-processing"
Nov 03 22:59:51.840160 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:51.840081    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b19d9f9-17b7-4140-a82c-4942facea4d2" containerName="test-curl-7266-37389c"
Nov 03 22:59:51.840160 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:51.840094    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fe7c171-abe2-428d-81e1-d13da5f0c376" containerName="payload-processing"
Nov 03 22:59:51.840160 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:51.840114    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2a90093-86e4-4bed-9892-31150b35ff48" containerName="payload-pre-processing"
Nov 03 22:59:51.844499 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:51.844476    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4a01f9-57fb584d-fn5pv"
Nov 03 22:59:51.862741 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:51.862706    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4a01f9-57fb584d-fn5pv"]
Nov 03 22:59:51.866082 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:51.866058    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-4a01f9-dockercfg-xxs2f\""
Nov 03 22:59:51.866262 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:51.866110    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-4a01f9\""
Nov 03 22:59:51.880425 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:51.880399    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4a01f9-6df8b8cb59-l6zlr"]
Nov 03 22:59:51.884803 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:51.884786    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4a01f9-6df8b8cb59-l6zlr"
Nov 03 22:59:51.901376 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:51.901348    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4a01f9-6df8b8cb59-l6zlr"]
Nov 03 22:59:51.920987 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:51.920949    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2a90093-86e4-4bed-9892-31150b35ff48-kube-api-access-vgl99\" (UniqueName: \"kubernetes.io/projected/e2a90093-86e4-4bed-9892-31150b35ff48-kube-api-access-vgl99\") pod \"e2a90093-86e4-4bed-9892-31150b35ff48\" (UID: \"e2a90093-86e4-4bed-9892-31150b35ff48\") "
Nov 03 22:59:51.921190 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:51.921077    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2a90093-86e4-4bed-9892-31150b35ff48-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2a90093-86e4-4bed-9892-31150b35ff48-plugins-config-volume\") pod \"e2a90093-86e4-4bed-9892-31150b35ff48\" (UID: \"e2a90093-86e4-4bed-9892-31150b35ff48\") "
Nov 03 22:59:51.921670 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:51.921634    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2a90093-86e4-4bed-9892-31150b35ff48-plugins-config-volume" pod "e2a90093-86e4-4bed-9892-31150b35ff48" (UID: "e2a90093-86e4-4bed-9892-31150b35ff48"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:59:51.921911 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:51.921883    4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2a90093-86e4-4bed-9892-31150b35ff48-plugins-config-volume\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:59:51.922003 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:51.921919    4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfb72051-95d9-406f-b200-e676af52c8ab-plugins-config-volume\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:59:51.922003 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:51.921934    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxnkt\" (UniqueName: \"kubernetes.io/projected/cfb72051-95d9-406f-b200-e676af52c8ab-kube-api-access-pxnkt\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:59:51.923291 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:51.923263    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2a90093-86e4-4bed-9892-31150b35ff48-kube-api-access-vgl99" pod "e2a90093-86e4-4bed-9892-31150b35ff48" (UID: "e2a90093-86e4-4bed-9892-31150b35ff48"). InnerVolumeSpecName "kube-api-access-vgl99". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:59:51.929580 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:51.929557    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:59:52.022990 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:52.022948    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e02c5146-c15c-4b43-9448-457efdde2d2c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4a01f9-6df8b8cb59-l6zlr\" (UID: \"e02c5146-c15c-4b43-9448-457efdde2d2c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4a01f9-6df8b8cb59-l6zlr"
Nov 03 22:59:52.023197 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:52.023009    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsld2\" (UniqueName: \"kubernetes.io/projected/e02c5146-c15c-4b43-9448-457efdde2d2c-kube-api-access-hsld2\") pod \"payload-pre-processing-e2e-ipp-b-4a01f9-6df8b8cb59-l6zlr\" (UID: \"e02c5146-c15c-4b43-9448-457efdde2d2c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4a01f9-6df8b8cb59-l6zlr"
Nov 03 22:59:52.023197 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:52.023086    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1b37d17-4abd-408b-a1dc-b89eefeb3393-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-4a01f9-57fb584d-fn5pv\" (UID: \"d1b37d17-4abd-408b-a1dc-b89eefeb3393\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4a01f9-57fb584d-fn5pv"
Nov 03 22:59:52.023197 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:52.023157    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwdsz\" (UniqueName: \"kubernetes.io/projected/d1b37d17-4abd-408b-a1dc-b89eefeb3393-kube-api-access-wwdsz\") pod \"payload-processing-e2e-ipp-b-4a01f9-57fb584d-fn5pv\" (UID: \"d1b37d17-4abd-408b-a1dc-b89eefeb3393\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4a01f9-57fb584d-fn5pv"
Nov 03 22:59:52.023455 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:52.023252    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgl99\" (UniqueName: \"kubernetes.io/projected/e2a90093-86e4-4bed-9892-31150b35ff48-kube-api-access-vgl99\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:59:52.124964 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:52.124925    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1b37d17-4abd-408b-a1dc-b89eefeb3393-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-4a01f9-57fb584d-fn5pv\" (UID: \"d1b37d17-4abd-408b-a1dc-b89eefeb3393\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4a01f9-57fb584d-fn5pv"
Nov 03 22:59:52.125156 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:52.125017    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwdsz\" (UniqueName: \"kubernetes.io/projected/d1b37d17-4abd-408b-a1dc-b89eefeb3393-kube-api-access-wwdsz\") pod \"payload-processing-e2e-ipp-b-4a01f9-57fb584d-fn5pv\" (UID: \"d1b37d17-4abd-408b-a1dc-b89eefeb3393\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4a01f9-57fb584d-fn5pv"
Nov 03 22:59:52.125156 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:52.125119    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e02c5146-c15c-4b43-9448-457efdde2d2c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4a01f9-6df8b8cb59-l6zlr\" (UID: \"e02c5146-c15c-4b43-9448-457efdde2d2c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4a01f9-6df8b8cb59-l6zlr"
Nov 03 22:59:52.125312 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:52.125154    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsld2\" (UniqueName: \"kubernetes.io/projected/e02c5146-c15c-4b43-9448-457efdde2d2c-kube-api-access-hsld2\") pod \"payload-pre-processing-e2e-ipp-b-4a01f9-6df8b8cb59-l6zlr\" (UID: \"e02c5146-c15c-4b43-9448-457efdde2d2c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4a01f9-6df8b8cb59-l6zlr"
Nov 03 22:59:52.126208 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:52.126179    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1b37d17-4abd-408b-a1dc-b89eefeb3393-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-4a01f9-57fb584d-fn5pv\" (UID: \"d1b37d17-4abd-408b-a1dc-b89eefeb3393\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4a01f9-57fb584d-fn5pv"
Nov 03 22:59:52.128453 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:52.126718    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e02c5146-c15c-4b43-9448-457efdde2d2c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4a01f9-6df8b8cb59-l6zlr\" (UID: \"e02c5146-c15c-4b43-9448-457efdde2d2c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4a01f9-6df8b8cb59-l6zlr"
Nov 03 22:59:52.148402 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:52.148363    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwdsz\" (UniqueName: \"kubernetes.io/projected/d1b37d17-4abd-408b-a1dc-b89eefeb3393-kube-api-access-wwdsz\") pod \"payload-processing-e2e-ipp-b-4a01f9-57fb584d-fn5pv\" (UID: \"d1b37d17-4abd-408b-a1dc-b89eefeb3393\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4a01f9-57fb584d-fn5pv"
Nov 03 22:59:52.148570 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:52.148460    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsld2\" (UniqueName: \"kubernetes.io/projected/e02c5146-c15c-4b43-9448-457efdde2d2c-kube-api-access-hsld2\") pod \"payload-pre-processing-e2e-ipp-b-4a01f9-6df8b8cb59-l6zlr\" (UID: \"e02c5146-c15c-4b43-9448-457efdde2d2c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4a01f9-6df8b8cb59-l6zlr"
Nov 03 22:59:52.165804 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:52.165775    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4a01f9-57fb584d-fn5pv"
Nov 03 22:59:52.206271 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:52.206239    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4a01f9-6df8b8cb59-l6zlr"
Nov 03 22:59:52.292070 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:52.291761    4431 generic.go:356] "Generic (PLEG): container finished" podID="e2a90093-86e4-4bed-9892-31150b35ff48" containerID="0dc27536e80b98489f5e1dafd8154c7feae26013a8b445c2af60a2cf9fcc657d" exitCode=0
Nov 03 22:59:52.292070 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:52.291822    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15e7a7-5467566b7-npkkl"
Nov 03 22:59:52.292070 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:52.291878    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15e7a7-5467566b7-npkkl" event={"ID":"e2a90093-86e4-4bed-9892-31150b35ff48","Type":"ContainerDied","Data":"0dc27536e80b98489f5e1dafd8154c7feae26013a8b445c2af60a2cf9fcc657d"}
Nov 03 22:59:52.292070 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:52.291944    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15e7a7-5467566b7-npkkl" event={"ID":"e2a90093-86e4-4bed-9892-31150b35ff48","Type":"ContainerDied","Data":"b8b803690ea3aa6237e8f689d77e7d7850b9566c1ae96d25e8a326a47ff2b169"}
Nov 03 22:59:52.292070 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:52.291966    4431 scope.go:122] "RemoveContainer" containerID="0dc27536e80b98489f5e1dafd8154c7feae26013a8b445c2af60a2cf9fcc657d"
Nov 03 22:59:52.294318 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:52.294182    4431 generic.go:356] "Generic (PLEG): container finished" podID="cfb72051-95d9-406f-b200-e676af52c8ab" containerID="08a2498654efb31d6dd7ab9367c9abf357856d7158fd8a1f5830936b8f327d7a" exitCode=0
Nov 03 22:59:52.294318 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:52.294251    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-15e7a7-79b4b8dbd4-pb92m"
Nov 03 22:59:52.294611 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:52.294522    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-15e7a7-79b4b8dbd4-pb92m" event={"ID":"cfb72051-95d9-406f-b200-e676af52c8ab","Type":"ContainerDied","Data":"08a2498654efb31d6dd7ab9367c9abf357856d7158fd8a1f5830936b8f327d7a"}
Nov 03 22:59:52.294611 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:52.294581    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-15e7a7-79b4b8dbd4-pb92m" event={"ID":"cfb72051-95d9-406f-b200-e676af52c8ab","Type":"ContainerDied","Data":"2bdd7e4efa77c8c47bf35029d24c7b24216710e5ee2fa20a0ebe4a59aa96e640"}
Nov 03 22:59:52.308641 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:52.308616    4431 scope.go:122] "RemoveContainer" containerID="0dc27536e80b98489f5e1dafd8154c7feae26013a8b445c2af60a2cf9fcc657d"
Nov 03 22:59:52.309041 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:59:52.309014    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dc27536e80b98489f5e1dafd8154c7feae26013a8b445c2af60a2cf9fcc657d\": container with ID starting with 0dc27536e80b98489f5e1dafd8154c7feae26013a8b445c2af60a2cf9fcc657d not found: ID does not exist" containerID="0dc27536e80b98489f5e1dafd8154c7feae26013a8b445c2af60a2cf9fcc657d"
Nov 03 22:59:52.309129 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:52.309048    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dc27536e80b98489f5e1dafd8154c7feae26013a8b445c2af60a2cf9fcc657d"} err="failed to get container status \"0dc27536e80b98489f5e1dafd8154c7feae26013a8b445c2af60a2cf9fcc657d\": rpc error: code = NotFound desc = could not find container \"0dc27536e80b98489f5e1dafd8154c7feae26013a8b445c2af60a2cf9fcc657d\": container with ID starting with 0dc27536e80b98489f5e1dafd8154c7feae26013a8b445c2af60a2cf9fcc657d not found: ID does not exist"
Nov 03 22:59:52.309129 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:52.309065    4431 scope.go:122] "RemoveContainer" containerID="08a2498654efb31d6dd7ab9367c9abf357856d7158fd8a1f5830936b8f327d7a"
Nov 03 22:59:52.327922 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:52.327895    4431 scope.go:122] "RemoveContainer" containerID="08a2498654efb31d6dd7ab9367c9abf357856d7158fd8a1f5830936b8f327d7a"
Nov 03 22:59:52.328322 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:59:52.328297    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08a2498654efb31d6dd7ab9367c9abf357856d7158fd8a1f5830936b8f327d7a\": container with ID starting with 08a2498654efb31d6dd7ab9367c9abf357856d7158fd8a1f5830936b8f327d7a not found: ID does not exist" containerID="08a2498654efb31d6dd7ab9367c9abf357856d7158fd8a1f5830936b8f327d7a"
Nov 03 22:59:52.328435 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:52.328331    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08a2498654efb31d6dd7ab9367c9abf357856d7158fd8a1f5830936b8f327d7a"} err="failed to get container status \"08a2498654efb31d6dd7ab9367c9abf357856d7158fd8a1f5830936b8f327d7a\": rpc error: code = NotFound desc = could not find container \"08a2498654efb31d6dd7ab9367c9abf357856d7158fd8a1f5830936b8f327d7a\": container with ID starting with 08a2498654efb31d6dd7ab9367c9abf357856d7158fd8a1f5830936b8f327d7a not found: ID does not exist"
Nov 03 22:59:52.345861 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:52.345799    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-15e7a7-79b4b8dbd4-pb92m"]
Nov 03 22:59:52.349559 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:52.349485    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-15e7a7-79b4b8dbd4-pb92m"]
Nov 03 22:59:52.381912 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:52.381876    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-15e7a7-5467566b7-npkkl"]
Nov 03 22:59:52.388333 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:52.388303    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-15e7a7-5467566b7-npkkl"]
Nov 03 22:59:52.477157 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:52.477123    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4a01f9-57fb584d-fn5pv"]
Nov 03 22:59:52.525793 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:52.525765    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4a01f9-6df8b8cb59-l6zlr"]
Nov 03 22:59:52.537749 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:59:52.537718    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode02c5146_c15c_4b43_9448_457efdde2d2c.slice/crio-b3387282691143981a5391bfd5c250e00769b512b30263b177c80a02b1aa8352 WatchSource:0}: Error finding container b3387282691143981a5391bfd5c250e00769b512b30263b177c80a02b1aa8352: Status 404 returned error can't find the container with id b3387282691143981a5391bfd5c250e00769b512b30263b177c80a02b1aa8352
Nov 03 22:59:53.305724 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:53.305688    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4a01f9-6df8b8cb59-l6zlr" event={"ID":"e02c5146-c15c-4b43-9448-457efdde2d2c","Type":"ContainerStarted","Data":"b9933420b8878d6068e77328e964d06b6eef60e5250ab71c43dc4c24698d1652"}
Nov 03 22:59:53.306161 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:53.305738    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4a01f9-6df8b8cb59-l6zlr" event={"ID":"e02c5146-c15c-4b43-9448-457efdde2d2c","Type":"ContainerStarted","Data":"b3387282691143981a5391bfd5c250e00769b512b30263b177c80a02b1aa8352"}
Nov 03 22:59:53.306161 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:53.305812    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4a01f9-6df8b8cb59-l6zlr"
Nov 03 22:59:53.307381 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:53.307357    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4a01f9-57fb584d-fn5pv" event={"ID":"d1b37d17-4abd-408b-a1dc-b89eefeb3393","Type":"ContainerStarted","Data":"64554b336c7280e5b3e62d9913efc83887509eb1469a448d547d8ffe28e2f853"}
Nov 03 22:59:53.307534 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:53.307399    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4a01f9-57fb584d-fn5pv" event={"ID":"d1b37d17-4abd-408b-a1dc-b89eefeb3393","Type":"ContainerStarted","Data":"844cd1779d3d94b194d76c028aef99097b0fb67293914d91c0f721025d5c12f0"}
Nov 03 22:59:53.307534 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:53.307491    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-4a01f9-57fb584d-fn5pv"
Nov 03 22:59:53.345257 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:53.345133    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4a01f9-6df8b8cb59-l6zlr" podStartSLOduration=2.34511817 podStartE2EDuration="2.34511817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:59:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:59:53.344381629 +0000 UTC m=+2817.370237488" watchObservedRunningTime="2026-11-03 22:59:53.34511817 +0000 UTC m=+2817.370974033"
Nov 03 22:59:53.385181 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:53.385121    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-4a01f9-57fb584d-fn5pv" podStartSLOduration=2.385102041 podStartE2EDuration="2.385102041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:59:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:59:53.382922561 +0000 UTC m=+2817.408778423" watchObservedRunningTime="2026-11-03 22:59:53.385102041 +0000 UTC m=+2817.410957907"
Nov 03 22:59:53.427664 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:53.427630    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:59:54.090664 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:54.090634    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfb72051-95d9-406f-b200-e676af52c8ab" path="/var/lib/kubelet/pods/cfb72051-95d9-406f-b200-e676af52c8ab/volumes"
Nov 03 22:59:54.091068 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:54.091054    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2a90093-86e4-4bed-9892-31150b35ff48" path="/var/lib/kubelet/pods/e2a90093-86e4-4bed-9892-31150b35ff48/volumes"
Nov 03 22:59:54.908843 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:54.908800    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-3920cb/ipp-route-3920cb-kserve-76df875dbf-rrqjq"]
Nov 03 22:59:54.915389 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:54.915361    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-3920cb/ipp-route-3920cb-kserve-76df875dbf-rrqjq"
Nov 03 22:59:54.942939 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:54.942907    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-3920cb\"/\"ipp-route-3920cb-kserve-self-signed-certs\""
Nov 03 22:59:54.943103 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:54.943054    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-3920cb\"/\"default-dockercfg-qthjv\""
Nov 03 22:59:54.943188 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:54.943140    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-3920cb\"/\"openshift-service-ca.crt\""
Nov 03 22:59:54.943269 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:54.943249    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-3920cb\"/\"kube-root-ca.crt\""
Nov 03 22:59:54.947021 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:54.946995    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-3920cb/ipp-route-3920cb-kserve-76df875dbf-rrqjq"]
Nov 03 22:59:54.964105 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:54.964072    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fc68969-ebce-4cfb-bdb0-8600510e931f-home\") pod \"ipp-route-3920cb-kserve-76df875dbf-rrqjq\" (UID: \"0fc68969-ebce-4cfb-bdb0-8600510e931f\") " pod="ai-tenant-e2e-ipp-a-3920cb/ipp-route-3920cb-kserve-76df875dbf-rrqjq"
Nov 03 22:59:54.964273 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:54.964109    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fc68969-ebce-4cfb-bdb0-8600510e931f-tls-certs\") pod \"ipp-route-3920cb-kserve-76df875dbf-rrqjq\" (UID: \"0fc68969-ebce-4cfb-bdb0-8600510e931f\") " pod="ai-tenant-e2e-ipp-a-3920cb/ipp-route-3920cb-kserve-76df875dbf-rrqjq"
Nov 03 22:59:54.964273 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:54.964138    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fc68969-ebce-4cfb-bdb0-8600510e931f-dshm\") pod \"ipp-route-3920cb-kserve-76df875dbf-rrqjq\" (UID: \"0fc68969-ebce-4cfb-bdb0-8600510e931f\") " pod="ai-tenant-e2e-ipp-a-3920cb/ipp-route-3920cb-kserve-76df875dbf-rrqjq"
Nov 03 22:59:54.964408 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:54.964311    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fc68969-ebce-4cfb-bdb0-8600510e931f-model-cache\") pod \"ipp-route-3920cb-kserve-76df875dbf-rrqjq\" (UID: \"0fc68969-ebce-4cfb-bdb0-8600510e931f\") " pod="ai-tenant-e2e-ipp-a-3920cb/ipp-route-3920cb-kserve-76df875dbf-rrqjq"
Nov 03 22:59:54.964502 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:54.964479    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv25f\" (UniqueName: \"kubernetes.io/projected/0fc68969-ebce-4cfb-bdb0-8600510e931f-kube-api-access-tv25f\") pod \"ipp-route-3920cb-kserve-76df875dbf-rrqjq\" (UID: \"0fc68969-ebce-4cfb-bdb0-8600510e931f\") " pod="ai-tenant-e2e-ipp-a-3920cb/ipp-route-3920cb-kserve-76df875dbf-rrqjq"
Nov 03 22:59:54.964603 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:54.964583    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fc68969-ebce-4cfb-bdb0-8600510e931f-tmp-dir\") pod \"ipp-route-3920cb-kserve-76df875dbf-rrqjq\" (UID: \"0fc68969-ebce-4cfb-bdb0-8600510e931f\") " pod="ai-tenant-e2e-ipp-a-3920cb/ipp-route-3920cb-kserve-76df875dbf-rrqjq"
Nov 03 22:59:54.998845 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:54.998806    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:59:55.065840 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:55.065808    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fc68969-ebce-4cfb-bdb0-8600510e931f-home\") pod \"ipp-route-3920cb-kserve-76df875dbf-rrqjq\" (UID: \"0fc68969-ebce-4cfb-bdb0-8600510e931f\") " pod="ai-tenant-e2e-ipp-a-3920cb/ipp-route-3920cb-kserve-76df875dbf-rrqjq"
Nov 03 22:59:55.066056 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:55.065852    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fc68969-ebce-4cfb-bdb0-8600510e931f-tls-certs\") pod \"ipp-route-3920cb-kserve-76df875dbf-rrqjq\" (UID: \"0fc68969-ebce-4cfb-bdb0-8600510e931f\") " pod="ai-tenant-e2e-ipp-a-3920cb/ipp-route-3920cb-kserve-76df875dbf-rrqjq"
Nov 03 22:59:55.066056 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:55.065882    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fc68969-ebce-4cfb-bdb0-8600510e931f-dshm\") pod \"ipp-route-3920cb-kserve-76df875dbf-rrqjq\" (UID: \"0fc68969-ebce-4cfb-bdb0-8600510e931f\") " pod="ai-tenant-e2e-ipp-a-3920cb/ipp-route-3920cb-kserve-76df875dbf-rrqjq"
Nov 03 22:59:55.066056 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:55.065930    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fc68969-ebce-4cfb-bdb0-8600510e931f-model-cache\") pod \"ipp-route-3920cb-kserve-76df875dbf-rrqjq\" (UID: \"0fc68969-ebce-4cfb-bdb0-8600510e931f\") " pod="ai-tenant-e2e-ipp-a-3920cb/ipp-route-3920cb-kserve-76df875dbf-rrqjq"
Nov 03 22:59:55.066268 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:55.066076    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv25f\" (UniqueName: \"kubernetes.io/projected/0fc68969-ebce-4cfb-bdb0-8600510e931f-kube-api-access-tv25f\") pod \"ipp-route-3920cb-kserve-76df875dbf-rrqjq\" (UID: \"0fc68969-ebce-4cfb-bdb0-8600510e931f\") " pod="ai-tenant-e2e-ipp-a-3920cb/ipp-route-3920cb-kserve-76df875dbf-rrqjq"
Nov 03 22:59:55.066268 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:55.066174    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fc68969-ebce-4cfb-bdb0-8600510e931f-tmp-dir\") pod \"ipp-route-3920cb-kserve-76df875dbf-rrqjq\" (UID: \"0fc68969-ebce-4cfb-bdb0-8600510e931f\") " pod="ai-tenant-e2e-ipp-a-3920cb/ipp-route-3920cb-kserve-76df875dbf-rrqjq"
Nov 03 22:59:55.066388 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:55.066363    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fc68969-ebce-4cfb-bdb0-8600510e931f-home\") pod \"ipp-route-3920cb-kserve-76df875dbf-rrqjq\" (UID: \"0fc68969-ebce-4cfb-bdb0-8600510e931f\") " pod="ai-tenant-e2e-ipp-a-3920cb/ipp-route-3920cb-kserve-76df875dbf-rrqjq"
Nov 03 22:59:55.066456 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:55.066425    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fc68969-ebce-4cfb-bdb0-8600510e931f-model-cache\") pod \"ipp-route-3920cb-kserve-76df875dbf-rrqjq\" (UID: \"0fc68969-ebce-4cfb-bdb0-8600510e931f\") " pod="ai-tenant-e2e-ipp-a-3920cb/ipp-route-3920cb-kserve-76df875dbf-rrqjq"
Nov 03 22:59:55.066679 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:55.066617    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fc68969-ebce-4cfb-bdb0-8600510e931f-tmp-dir\") pod \"ipp-route-3920cb-kserve-76df875dbf-rrqjq\" (UID: \"0fc68969-ebce-4cfb-bdb0-8600510e931f\") " pod="ai-tenant-e2e-ipp-a-3920cb/ipp-route-3920cb-kserve-76df875dbf-rrqjq"
Nov 03 22:59:55.068666 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:55.068641    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fc68969-ebce-4cfb-bdb0-8600510e931f-dshm\") pod \"ipp-route-3920cb-kserve-76df875dbf-rrqjq\" (UID: \"0fc68969-ebce-4cfb-bdb0-8600510e931f\") " pod="ai-tenant-e2e-ipp-a-3920cb/ipp-route-3920cb-kserve-76df875dbf-rrqjq"
Nov 03 22:59:55.068824 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:55.068806    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fc68969-ebce-4cfb-bdb0-8600510e931f-tls-certs\") pod \"ipp-route-3920cb-kserve-76df875dbf-rrqjq\" (UID: \"0fc68969-ebce-4cfb-bdb0-8600510e931f\") " pod="ai-tenant-e2e-ipp-a-3920cb/ipp-route-3920cb-kserve-76df875dbf-rrqjq"
Nov 03 22:59:55.098211 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:55.098182    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv25f\" (UniqueName: \"kubernetes.io/projected/0fc68969-ebce-4cfb-bdb0-8600510e931f-kube-api-access-tv25f\") pod \"ipp-route-3920cb-kserve-76df875dbf-rrqjq\" (UID: \"0fc68969-ebce-4cfb-bdb0-8600510e931f\") " pod="ai-tenant-e2e-ipp-a-3920cb/ipp-route-3920cb-kserve-76df875dbf-rrqjq"
Nov 03 22:59:55.159721 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:55.159632    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:59:55.233273 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:55.231510    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-3920cb/ipp-route-3920cb-kserve-76df875dbf-rrqjq"
Nov 03 22:59:55.564727 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:55.564700    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-3920cb/ipp-route-3920cb-kserve-76df875dbf-rrqjq"]
Nov 03 22:59:55.566339 ip-10-0-139-73 kubenswrapper[4431]: W1103 22:59:55.566313    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0fc68969_ebce_4cfb_bdb0_8600510e931f.slice/crio-32e68125fda8101c5b8c86c6c326526a4fb5745234d048543ddbe1b6d1f281ce WatchSource:0}: Error finding container 32e68125fda8101c5b8c86c6c326526a4fb5745234d048543ddbe1b6d1f281ce: Status 404 returned error can't find the container with id 32e68125fda8101c5b8c86c6c326526a4fb5745234d048543ddbe1b6d1f281ce
Nov 03 22:59:56.353808 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:56.353766    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-3920cb/ipp-route-3920cb-kserve-76df875dbf-rrqjq" event={"ID":"0fc68969-ebce-4cfb-bdb0-8600510e931f","Type":"ContainerStarted","Data":"4585bf084388c39eeaa35cf1e4f2a9043e63ba6917678722fb9841ed01fee112"}
Nov 03 22:59:56.354256 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:56.353821    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-3920cb/ipp-route-3920cb-kserve-76df875dbf-rrqjq" event={"ID":"0fc68969-ebce-4cfb-bdb0-8600510e931f","Type":"ContainerStarted","Data":"32e68125fda8101c5b8c86c6c326526a4fb5745234d048543ddbe1b6d1f281ce"}
Nov 03 22:59:56.862268 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:56.862198    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:59:57.239241 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:57.232101    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-3920cb/ipp-route-3920cb-kserve-76df875dbf-rrqjq"
Nov 03 22:59:58.105414 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:58.105376    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:59:58.633887 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:58.633848    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:59:58.884752 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:58.884661    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:59:58.996010 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:58.995965    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w5-b023a2/auth-test-model-b023a2-kserve-7f85457dc7-pvx2r" podUID="1ba07c9c-1005-4d6f-b0af-ac5463bf588f" containerName="main" containerID="cri-o://6aee4ccf41d59433d93452cd39b692cfd9ac057e2d808bed313f7d05f81e789a" gracePeriod=45
Nov 03 22:59:59.306607 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:59.306573    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:59:59.333301 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:59.326919    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-b023a2/auth-test-model-b023a2-kserve-7f85457dc7-pvx2r"
Nov 03 22:59:59.375746 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:59.375655    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-3920cb/ipp-route-3920cb-kserve-76df875dbf-rrqjq" podStartSLOduration=5.375625596 podStartE2EDuration="5.375625596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:59:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:59:56.433237745 +0000 UTC m=+2820.459093620" watchObservedRunningTime="2026-11-03 22:59:59.375625596 +0000 UTC m=+2823.401481480"
Nov 03 22:59:59.426857 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:59.420006    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ba07c9c-1005-4d6f-b0af-ac5463bf588f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ba07c9c-1005-4d6f-b0af-ac5463bf588f-tmp-dir\") pod \"1ba07c9c-1005-4d6f-b0af-ac5463bf588f\" (UID: \"1ba07c9c-1005-4d6f-b0af-ac5463bf588f\") "
Nov 03 22:59:59.426857 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:59.420082    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ba07c9c-1005-4d6f-b0af-ac5463bf588f-kube-api-access-tzs6f\" (UniqueName: \"kubernetes.io/projected/1ba07c9c-1005-4d6f-b0af-ac5463bf588f-kube-api-access-tzs6f\") pod \"1ba07c9c-1005-4d6f-b0af-ac5463bf588f\" (UID: \"1ba07c9c-1005-4d6f-b0af-ac5463bf588f\") "
Nov 03 22:59:59.426857 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:59.420130    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ba07c9c-1005-4d6f-b0af-ac5463bf588f-home\" (UniqueName: \"kubernetes.io/empty-dir/1ba07c9c-1005-4d6f-b0af-ac5463bf588f-home\") pod \"1ba07c9c-1005-4d6f-b0af-ac5463bf588f\" (UID: \"1ba07c9c-1005-4d6f-b0af-ac5463bf588f\") "
Nov 03 22:59:59.426857 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:59.420194    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ba07c9c-1005-4d6f-b0af-ac5463bf588f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ba07c9c-1005-4d6f-b0af-ac5463bf588f-dshm\") pod \"1ba07c9c-1005-4d6f-b0af-ac5463bf588f\" (UID: \"1ba07c9c-1005-4d6f-b0af-ac5463bf588f\") "
Nov 03 22:59:59.426857 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:59.420284    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ba07c9c-1005-4d6f-b0af-ac5463bf588f-tls-certs\" (UniqueName: \"kubernetes.io/secret/1ba07c9c-1005-4d6f-b0af-ac5463bf588f-tls-certs\") pod \"1ba07c9c-1005-4d6f-b0af-ac5463bf588f\" (UID: \"1ba07c9c-1005-4d6f-b0af-ac5463bf588f\") "
Nov 03 22:59:59.426857 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:59.420433    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ba07c9c-1005-4d6f-b0af-ac5463bf588f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ba07c9c-1005-4d6f-b0af-ac5463bf588f-model-cache\") pod \"1ba07c9c-1005-4d6f-b0af-ac5463bf588f\" (UID: \"1ba07c9c-1005-4d6f-b0af-ac5463bf588f\") "
Nov 03 22:59:59.426857 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:59.421354    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ba07c9c-1005-4d6f-b0af-ac5463bf588f-model-cache" pod "1ba07c9c-1005-4d6f-b0af-ac5463bf588f" (UID: "1ba07c9c-1005-4d6f-b0af-ac5463bf588f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:59:59.426857 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:59.421603    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ba07c9c-1005-4d6f-b0af-ac5463bf588f-tmp-dir" pod "1ba07c9c-1005-4d6f-b0af-ac5463bf588f" (UID: "1ba07c9c-1005-4d6f-b0af-ac5463bf588f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:59:59.426857 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:59.422426    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ba07c9c-1005-4d6f-b0af-ac5463bf588f-home" pod "1ba07c9c-1005-4d6f-b0af-ac5463bf588f" (UID: "1ba07c9c-1005-4d6f-b0af-ac5463bf588f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:59:59.426857 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:59.425027    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ba07c9c-1005-4d6f-b0af-ac5463bf588f-kube-api-access-tzs6f" pod "1ba07c9c-1005-4d6f-b0af-ac5463bf588f" (UID: "1ba07c9c-1005-4d6f-b0af-ac5463bf588f"). InnerVolumeSpecName "kube-api-access-tzs6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:59:59.429386 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:59.427933    4431 generic.go:356] "Generic (PLEG): container finished" podID="1ba07c9c-1005-4d6f-b0af-ac5463bf588f" containerID="6aee4ccf41d59433d93452cd39b692cfd9ac057e2d808bed313f7d05f81e789a" exitCode=0
Nov 03 22:59:59.429386 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:59.428127    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-b023a2/auth-test-model-b023a2-kserve-7f85457dc7-pvx2r" event={"ID":"1ba07c9c-1005-4d6f-b0af-ac5463bf588f","Type":"ContainerDied","Data":"6aee4ccf41d59433d93452cd39b692cfd9ac057e2d808bed313f7d05f81e789a"}
Nov 03 22:59:59.429386 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:59.428161    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-b023a2/auth-test-model-b023a2-kserve-7f85457dc7-pvx2r" event={"ID":"1ba07c9c-1005-4d6f-b0af-ac5463bf588f","Type":"ContainerDied","Data":"92c5fdafeccdad0842116903fff7d0f6da3790a1961e5e90d30d186ec088ed61"}
Nov 03 22:59:59.429386 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:59.428188    4431 scope.go:122] "RemoveContainer" containerID="6aee4ccf41d59433d93452cd39b692cfd9ac057e2d808bed313f7d05f81e789a"
Nov 03 22:59:59.429386 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:59.428359    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-b023a2/auth-test-model-b023a2-kserve-7f85457dc7-pvx2r"
Nov 03 22:59:59.443152 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:59.442821    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ba07c9c-1005-4d6f-b0af-ac5463bf588f-tls-certs" pod "1ba07c9c-1005-4d6f-b0af-ac5463bf588f" (UID: "1ba07c9c-1005-4d6f-b0af-ac5463bf588f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:59:59.455328 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:59.455126    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ba07c9c-1005-4d6f-b0af-ac5463bf588f-dshm" pod "1ba07c9c-1005-4d6f-b0af-ac5463bf588f" (UID: "1ba07c9c-1005-4d6f-b0af-ac5463bf588f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:59:59.481497 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:59.481461    4431 scope.go:122] "RemoveContainer" containerID="6aee4ccf41d59433d93452cd39b692cfd9ac057e2d808bed313f7d05f81e789a"
Nov 03 22:59:59.481898 ip-10-0-139-73 kubenswrapper[4431]: E1103 22:59:59.481867    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6aee4ccf41d59433d93452cd39b692cfd9ac057e2d808bed313f7d05f81e789a\": container with ID starting with 6aee4ccf41d59433d93452cd39b692cfd9ac057e2d808bed313f7d05f81e789a not found: ID does not exist" containerID="6aee4ccf41d59433d93452cd39b692cfd9ac057e2d808bed313f7d05f81e789a"
Nov 03 22:59:59.481996 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:59.481912    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6aee4ccf41d59433d93452cd39b692cfd9ac057e2d808bed313f7d05f81e789a"} err="failed to get container status \"6aee4ccf41d59433d93452cd39b692cfd9ac057e2d808bed313f7d05f81e789a\": rpc error: code = NotFound desc = could not find container \"6aee4ccf41d59433d93452cd39b692cfd9ac057e2d808bed313f7d05f81e789a\": container with ID starting with 6aee4ccf41d59433d93452cd39b692cfd9ac057e2d808bed313f7d05f81e789a not found: ID does not exist"
Nov 03 22:59:59.522785 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:59.522753    4431 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ba07c9c-1005-4d6f-b0af-ac5463bf588f-dshm\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:59:59.522785 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:59.522782    4431 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ba07c9c-1005-4d6f-b0af-ac5463bf588f-tls-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:59:59.522785 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:59.522791    4431 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ba07c9c-1005-4d6f-b0af-ac5463bf588f-model-cache\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:59:59.523029 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:59.522800    4431 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ba07c9c-1005-4d6f-b0af-ac5463bf588f-tmp-dir\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:59:59.523029 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:59.522809    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzs6f\" (UniqueName: \"kubernetes.io/projected/1ba07c9c-1005-4d6f-b0af-ac5463bf588f-kube-api-access-tzs6f\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:59:59.523029 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:59.522817    4431 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ba07c9c-1005-4d6f-b0af-ac5463bf588f-home\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 22:59:59.745263 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:59.745231    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w6-2a2eca/rate-test-model-2a2eca-kserve-7bf8dbc765-xm2vl"]
Nov 03 22:59:59.746255 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:59.746208    4431 cpu_manager.go:414] "containerMap: removing container" podUID="1ba07c9c-1005-4d6f-b0af-ac5463bf588f" containerName="main"
Nov 03 22:59:59.746255 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:59.746253    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ba07c9c-1005-4d6f-b0af-ac5463bf588f" containerName="main"
Nov 03 22:59:59.746441 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:59.746428    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ba07c9c-1005-4d6f-b0af-ac5463bf588f" containerName="main"
Nov 03 22:59:59.752046 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:59.752013    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-2a2eca/rate-test-model-2a2eca-kserve-7bf8dbc765-xm2vl"
Nov 03 22:59:59.778015 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:59.776232    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w6-2a2eca\"/\"rate-test-model-2a2eca-kserve-self-signed-certs\""
Nov 03 22:59:59.778015 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:59.776332    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w6-2a2eca\"/\"kube-root-ca.crt\""
Nov 03 22:59:59.778015 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:59.776600    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w6-2a2eca\"/\"default-dockercfg-6fnmp\""
Nov 03 22:59:59.781322 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:59.781269    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w6-2a2eca/rate-test-model-2a2eca-kserve-7bf8dbc765-xm2vl"]
Nov 03 22:59:59.783321 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:59.783292    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:59:59.787898 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:59.787379    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w6-2a2eca\"/\"openshift-service-ca.crt\""
Nov 03 22:59:59.808125 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:59.808075    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-b023a2/auth-test-model-b023a2-kserve-7f85457dc7-pvx2r"]
Nov 03 22:59:59.829623 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:59.829584    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w5-b023a2/auth-test-model-b023a2-kserve-7f85457dc7-pvx2r"]
Nov 03 22:59:59.831763 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:59.831730    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4e5734b-ac12-4292-beb0-77d4aa2c08d6-tmp-dir\") pod \"rate-test-model-2a2eca-kserve-7bf8dbc765-xm2vl\" (UID: \"e4e5734b-ac12-4292-beb0-77d4aa2c08d6\") " pod="ai-tenant-e2e-shared-a-w6-2a2eca/rate-test-model-2a2eca-kserve-7bf8dbc765-xm2vl"
Nov 03 22:59:59.831919 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:59.831784    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4e5734b-ac12-4292-beb0-77d4aa2c08d6-tls-certs\") pod \"rate-test-model-2a2eca-kserve-7bf8dbc765-xm2vl\" (UID: \"e4e5734b-ac12-4292-beb0-77d4aa2c08d6\") " pod="ai-tenant-e2e-shared-a-w6-2a2eca/rate-test-model-2a2eca-kserve-7bf8dbc765-xm2vl"
Nov 03 22:59:59.831919 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:59.831828    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4e5734b-ac12-4292-beb0-77d4aa2c08d6-dshm\") pod \"rate-test-model-2a2eca-kserve-7bf8dbc765-xm2vl\" (UID: \"e4e5734b-ac12-4292-beb0-77d4aa2c08d6\") " pod="ai-tenant-e2e-shared-a-w6-2a2eca/rate-test-model-2a2eca-kserve-7bf8dbc765-xm2vl"
Nov 03 22:59:59.832058 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:59.831925    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p9v5\" (UniqueName: \"kubernetes.io/projected/e4e5734b-ac12-4292-beb0-77d4aa2c08d6-kube-api-access-7p9v5\") pod \"rate-test-model-2a2eca-kserve-7bf8dbc765-xm2vl\" (UID: \"e4e5734b-ac12-4292-beb0-77d4aa2c08d6\") " pod="ai-tenant-e2e-shared-a-w6-2a2eca/rate-test-model-2a2eca-kserve-7bf8dbc765-xm2vl"
Nov 03 22:59:59.832058 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:59.832026    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4e5734b-ac12-4292-beb0-77d4aa2c08d6-model-cache\") pod \"rate-test-model-2a2eca-kserve-7bf8dbc765-xm2vl\" (UID: \"e4e5734b-ac12-4292-beb0-77d4aa2c08d6\") " pod="ai-tenant-e2e-shared-a-w6-2a2eca/rate-test-model-2a2eca-kserve-7bf8dbc765-xm2vl"
Nov 03 22:59:59.832174 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:59.832111    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4e5734b-ac12-4292-beb0-77d4aa2c08d6-home\") pod \"rate-test-model-2a2eca-kserve-7bf8dbc765-xm2vl\" (UID: \"e4e5734b-ac12-4292-beb0-77d4aa2c08d6\") " pod="ai-tenant-e2e-shared-a-w6-2a2eca/rate-test-model-2a2eca-kserve-7bf8dbc765-xm2vl"
Nov 03 22:59:59.933657 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:59.933599    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4e5734b-ac12-4292-beb0-77d4aa2c08d6-model-cache\") pod \"rate-test-model-2a2eca-kserve-7bf8dbc765-xm2vl\" (UID: \"e4e5734b-ac12-4292-beb0-77d4aa2c08d6\") " pod="ai-tenant-e2e-shared-a-w6-2a2eca/rate-test-model-2a2eca-kserve-7bf8dbc765-xm2vl"
Nov 03 22:59:59.933860 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:59.933717    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4e5734b-ac12-4292-beb0-77d4aa2c08d6-home\") pod \"rate-test-model-2a2eca-kserve-7bf8dbc765-xm2vl\" (UID: \"e4e5734b-ac12-4292-beb0-77d4aa2c08d6\") " pod="ai-tenant-e2e-shared-a-w6-2a2eca/rate-test-model-2a2eca-kserve-7bf8dbc765-xm2vl"
Nov 03 22:59:59.933860 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:59.933843    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4e5734b-ac12-4292-beb0-77d4aa2c08d6-tmp-dir\") pod \"rate-test-model-2a2eca-kserve-7bf8dbc765-xm2vl\" (UID: \"e4e5734b-ac12-4292-beb0-77d4aa2c08d6\") " pod="ai-tenant-e2e-shared-a-w6-2a2eca/rate-test-model-2a2eca-kserve-7bf8dbc765-xm2vl"
Nov 03 22:59:59.933963 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:59.933880    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4e5734b-ac12-4292-beb0-77d4aa2c08d6-tls-certs\") pod \"rate-test-model-2a2eca-kserve-7bf8dbc765-xm2vl\" (UID: \"e4e5734b-ac12-4292-beb0-77d4aa2c08d6\") " pod="ai-tenant-e2e-shared-a-w6-2a2eca/rate-test-model-2a2eca-kserve-7bf8dbc765-xm2vl"
Nov 03 22:59:59.933963 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:59.933919    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4e5734b-ac12-4292-beb0-77d4aa2c08d6-dshm\") pod \"rate-test-model-2a2eca-kserve-7bf8dbc765-xm2vl\" (UID: \"e4e5734b-ac12-4292-beb0-77d4aa2c08d6\") " pod="ai-tenant-e2e-shared-a-w6-2a2eca/rate-test-model-2a2eca-kserve-7bf8dbc765-xm2vl"
Nov 03 22:59:59.934071 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:59.934013    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p9v5\" (UniqueName: \"kubernetes.io/projected/e4e5734b-ac12-4292-beb0-77d4aa2c08d6-kube-api-access-7p9v5\") pod \"rate-test-model-2a2eca-kserve-7bf8dbc765-xm2vl\" (UID: \"e4e5734b-ac12-4292-beb0-77d4aa2c08d6\") " pod="ai-tenant-e2e-shared-a-w6-2a2eca/rate-test-model-2a2eca-kserve-7bf8dbc765-xm2vl"
Nov 03 22:59:59.934904 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:59.934873    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4e5734b-ac12-4292-beb0-77d4aa2c08d6-model-cache\") pod \"rate-test-model-2a2eca-kserve-7bf8dbc765-xm2vl\" (UID: \"e4e5734b-ac12-4292-beb0-77d4aa2c08d6\") " pod="ai-tenant-e2e-shared-a-w6-2a2eca/rate-test-model-2a2eca-kserve-7bf8dbc765-xm2vl"
Nov 03 22:59:59.935162 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:59.935142    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4e5734b-ac12-4292-beb0-77d4aa2c08d6-home\") pod \"rate-test-model-2a2eca-kserve-7bf8dbc765-xm2vl\" (UID: \"e4e5734b-ac12-4292-beb0-77d4aa2c08d6\") " pod="ai-tenant-e2e-shared-a-w6-2a2eca/rate-test-model-2a2eca-kserve-7bf8dbc765-xm2vl"
Nov 03 22:59:59.935443 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:59.935419    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4e5734b-ac12-4292-beb0-77d4aa2c08d6-tmp-dir\") pod \"rate-test-model-2a2eca-kserve-7bf8dbc765-xm2vl\" (UID: \"e4e5734b-ac12-4292-beb0-77d4aa2c08d6\") " pod="ai-tenant-e2e-shared-a-w6-2a2eca/rate-test-model-2a2eca-kserve-7bf8dbc765-xm2vl"
Nov 03 22:59:59.938121 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:59.938097    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4e5734b-ac12-4292-beb0-77d4aa2c08d6-dshm\") pod \"rate-test-model-2a2eca-kserve-7bf8dbc765-xm2vl\" (UID: \"e4e5734b-ac12-4292-beb0-77d4aa2c08d6\") " pod="ai-tenant-e2e-shared-a-w6-2a2eca/rate-test-model-2a2eca-kserve-7bf8dbc765-xm2vl"
Nov 03 22:59:59.938459 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:59.938438    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4e5734b-ac12-4292-beb0-77d4aa2c08d6-tls-certs\") pod \"rate-test-model-2a2eca-kserve-7bf8dbc765-xm2vl\" (UID: \"e4e5734b-ac12-4292-beb0-77d4aa2c08d6\") " pod="ai-tenant-e2e-shared-a-w6-2a2eca/rate-test-model-2a2eca-kserve-7bf8dbc765-xm2vl"
Nov 03 22:59:59.942194 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:59.942166    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 22:59:59.957964 ip-10-0-139-73 kubenswrapper[4431]: I1103 22:59:59.957931    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p9v5\" (UniqueName: \"kubernetes.io/projected/e4e5734b-ac12-4292-beb0-77d4aa2c08d6-kube-api-access-7p9v5\") pod \"rate-test-model-2a2eca-kserve-7bf8dbc765-xm2vl\" (UID: \"e4e5734b-ac12-4292-beb0-77d4aa2c08d6\") " pod="ai-tenant-e2e-shared-a-w6-2a2eca/rate-test-model-2a2eca-kserve-7bf8dbc765-xm2vl"
Nov 03 23:00:00.092286 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.084466    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-2a2eca/rate-test-model-2a2eca-kserve-7bf8dbc765-xm2vl"
Nov 03 23:00:00.099491 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.099454    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ba07c9c-1005-4d6f-b0af-ac5463bf588f" path="/var/lib/kubelet/pods/1ba07c9c-1005-4d6f-b0af-ac5463bf588f/volumes"
Nov 03 23:00:00.215765 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.214757    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w5-b023a2/auth-test-model-b023a2-kserve-7f85457dc7-pvx2r" podUID="1ba07c9c-1005-4d6f-b0af-ac5463bf588f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.149:8000/ready\": context deadline exceeded"
Nov 03 23:00:00.230772 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.225630    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-23d281-29895780-cbhhq"]
Nov 03 23:00:00.246273 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.244604    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-23d281-29895780-cbhhq"
Nov 03 23:00:00.265341 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.265308    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5187e1-29895780-j9x65"]
Nov 03 23:00:00.303524 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.303495    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-23d281-29895780-cbhhq"]
Nov 03 23:00:00.303905 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.303884    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e19850-29895780-z748r"]
Nov 03 23:00:00.306294 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.303829    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5187e1-29895780-j9x65"
Nov 03 23:00:00.322679 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.322635    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\""
Nov 03 23:00:00.342811 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.340828    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c362bb2-ed25-42d4-8907-7ea5985b3df4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-23d281-29895780-cbhhq\" (UID: \"1c362bb2-ed25-42d4-8907-7ea5985b3df4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-23d281-29895780-cbhhq"
Nov 03 23:00:00.342811 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.340998    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d57hd\" (UniqueName: \"kubernetes.io/projected/1c362bb2-ed25-42d4-8907-7ea5985b3df4-kube-api-access-d57hd\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-23d281-29895780-cbhhq\" (UID: \"1c362bb2-ed25-42d4-8907-7ea5985b3df4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-23d281-29895780-cbhhq"
Nov 03 23:00:00.346869 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.346393    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5187e1-29895780-j9x65"]
Nov 03 23:00:00.346869 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.346447    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-50d738-29895780-ckqtt"]
Nov 03 23:00:00.346869 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.346485    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e19850-29895780-z748r"
Nov 03 23:00:00.352315 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.352292    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895780-dbslb"]
Nov 03 23:00:00.353439 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.353414    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-50d738-29895780-ckqtt"
Nov 03 23:00:00.358164 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.358144    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e19850-29895780-z748r"]
Nov 03 23:00:00.362020 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.362004    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895780-dbtlc"]
Nov 03 23:00:00.363291 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.358535    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895780-dbslb"
Nov 03 23:00:00.369129 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.369111    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-50d738-29895780-ckqtt"]
Nov 03 23:00:00.369376 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.369355    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-ff80c0-29895780-srjhc"]
Nov 03 23:00:00.371484 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.371277    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895780-dbtlc"
Nov 03 23:00:00.381503 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.381395    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895780-dbtlc"]
Nov 03 23:00:00.381615 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.381548    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3920cb-29895780-bhr6s"]
Nov 03 23:00:00.381784 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.381469    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-ff80c0-29895780-srjhc"
Nov 03 23:00:00.387886 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.387860    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-ff80c0-29895780-srjhc"]
Nov 03 23:00:00.393687 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.393392    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3920cb-29895780-bhr6s"
Nov 03 23:00:00.398532 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.398506    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895780-dbslb"]
Nov 03 23:00:00.405131 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.405102    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3920cb-29895780-bhr6s"]
Nov 03 23:00:00.443706 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.443372    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdjcc\" (UniqueName: \"kubernetes.io/projected/becaf054-eec7-43bb-8443-8d5052e5859a-kube-api-access-qdjcc\") pod \"maas-api-key-cleanup-e2e-worker-w3-5187e1-29895780-j9x65\" (UID: \"becaf054-eec7-43bb-8443-8d5052e5859a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5187e1-29895780-j9x65"
Nov 03 23:00:00.443706 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.443416    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrqbx\" (UniqueName: \"kubernetes.io/projected/b6d9effb-0fd1-4443-b1ee-82600bd004ff-kube-api-access-rrqbx\") pod \"maas-api-key-cleanup-e2e-worker-w1-e19850-29895780-z748r\" (UID: \"b6d9effb-0fd1-4443-b1ee-82600bd004ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e19850-29895780-z748r"
Nov 03 23:00:00.443706 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.443452    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6d9effb-0fd1-4443-b1ee-82600bd004ff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e19850-29895780-z748r\" (UID: \"b6d9effb-0fd1-4443-b1ee-82600bd004ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e19850-29895780-z748r"
Nov 03 23:00:00.443706 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.443487    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bfb8b9d-1ad7-4c94-96ab-25f74946da5b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-50d738-29895780-ckqtt\" (UID: \"8bfb8b9d-1ad7-4c94-96ab-25f74946da5b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-50d738-29895780-ckqtt"
Nov 03 23:00:00.443706 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.443548    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d57hd\" (UniqueName: \"kubernetes.io/projected/1c362bb2-ed25-42d4-8907-7ea5985b3df4-kube-api-access-d57hd\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-23d281-29895780-cbhhq\" (UID: \"1c362bb2-ed25-42d4-8907-7ea5985b3df4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-23d281-29895780-cbhhq"
Nov 03 23:00:00.443706 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.443600    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04763bf4-4b53-41a0-a47d-72d46f1f9563-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-ff80c0-29895780-srjhc\" (UID: \"04763bf4-4b53-41a0-a47d-72d46f1f9563\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-ff80c0-29895780-srjhc"
Nov 03 23:00:00.443706 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.443638    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/291a5765-87a6-46b2-88f4-1d4cd9056fab-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895780-dbtlc\" (UID: \"291a5765-87a6-46b2-88f4-1d4cd9056fab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895780-dbtlc"
Nov 03 23:00:00.444124 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.443729    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f01190d-0246-487f-a9af-54dab722b3e3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-3920cb-29895780-bhr6s\" (UID: \"1f01190d-0246-487f-a9af-54dab722b3e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3920cb-29895780-bhr6s"
Nov 03 23:00:00.444124 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.443758    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcpgp\" (UniqueName: \"kubernetes.io/projected/04763bf4-4b53-41a0-a47d-72d46f1f9563-kube-api-access-pcpgp\") pod \"maas-api-key-cleanup-e2e-ipp-b-ff80c0-29895780-srjhc\" (UID: \"04763bf4-4b53-41a0-a47d-72d46f1f9563\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-ff80c0-29895780-srjhc"
Nov 03 23:00:00.444124 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.443787    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cm9g\" (UniqueName: \"kubernetes.io/projected/9f6c89a6-83ae-4e3a-b10b-6da898e4857b-kube-api-access-6cm9g\") pod \"maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895780-dbslb\" (UID: \"9f6c89a6-83ae-4e3a-b10b-6da898e4857b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895780-dbslb"
Nov 03 23:00:00.444124 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.443824    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qg4pb\" (UniqueName: \"kubernetes.io/projected/8bfb8b9d-1ad7-4c94-96ab-25f74946da5b-kube-api-access-qg4pb\") pod \"maas-api-key-cleanup-e2e-worker-w4-50d738-29895780-ckqtt\" (UID: \"8bfb8b9d-1ad7-4c94-96ab-25f74946da5b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-50d738-29895780-ckqtt"
Nov 03 23:00:00.444124 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.443871    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f6c89a6-83ae-4e3a-b10b-6da898e4857b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895780-dbslb\" (UID: \"9f6c89a6-83ae-4e3a-b10b-6da898e4857b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895780-dbslb"
Nov 03 23:00:00.444124 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.443913    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/becaf054-eec7-43bb-8443-8d5052e5859a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-5187e1-29895780-j9x65\" (UID: \"becaf054-eec7-43bb-8443-8d5052e5859a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5187e1-29895780-j9x65"
Nov 03 23:00:00.444124 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.444035    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c362bb2-ed25-42d4-8907-7ea5985b3df4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-23d281-29895780-cbhhq\" (UID: \"1c362bb2-ed25-42d4-8907-7ea5985b3df4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-23d281-29895780-cbhhq"
Nov 03 23:00:00.444124 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.444097    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsd5f\" (UniqueName: \"kubernetes.io/projected/1f01190d-0246-487f-a9af-54dab722b3e3-kube-api-access-zsd5f\") pod \"maas-api-key-cleanup-e2e-ipp-a-3920cb-29895780-bhr6s\" (UID: \"1f01190d-0246-487f-a9af-54dab722b3e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3920cb-29895780-bhr6s"
Nov 03 23:00:00.444563 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.444140    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxl56\" (UniqueName: \"kubernetes.io/projected/291a5765-87a6-46b2-88f4-1d4cd9056fab-kube-api-access-lxl56\") pod \"maas-api-key-cleanup-29895780-dbtlc\" (UID: \"291a5765-87a6-46b2-88f4-1d4cd9056fab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895780-dbtlc"
Nov 03 23:00:00.445786 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.445760    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c362bb2-ed25-42d4-8907-7ea5985b3df4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-23d281-29895780-cbhhq\" (UID: \"1c362bb2-ed25-42d4-8907-7ea5985b3df4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-23d281-29895780-cbhhq"
Nov 03 23:00:00.457493 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.453664    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w6-2a2eca/rate-test-model-2a2eca-kserve-7bf8dbc765-xm2vl"]
Nov 03 23:00:00.479124 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.478478    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d57hd\" (UniqueName: \"kubernetes.io/projected/1c362bb2-ed25-42d4-8907-7ea5985b3df4-kube-api-access-d57hd\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-23d281-29895780-cbhhq\" (UID: \"1c362bb2-ed25-42d4-8907-7ea5985b3df4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-23d281-29895780-cbhhq"
Nov 03 23:00:00.545001 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.544976    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f01190d-0246-487f-a9af-54dab722b3e3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-3920cb-29895780-bhr6s\" (UID: \"1f01190d-0246-487f-a9af-54dab722b3e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3920cb-29895780-bhr6s"
Nov 03 23:00:00.545079 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.545012    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcpgp\" (UniqueName: \"kubernetes.io/projected/04763bf4-4b53-41a0-a47d-72d46f1f9563-kube-api-access-pcpgp\") pod \"maas-api-key-cleanup-e2e-ipp-b-ff80c0-29895780-srjhc\" (UID: \"04763bf4-4b53-41a0-a47d-72d46f1f9563\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-ff80c0-29895780-srjhc"
Nov 03 23:00:00.545079 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.545031    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cm9g\" (UniqueName: \"kubernetes.io/projected/9f6c89a6-83ae-4e3a-b10b-6da898e4857b-kube-api-access-6cm9g\") pod \"maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895780-dbslb\" (UID: \"9f6c89a6-83ae-4e3a-b10b-6da898e4857b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895780-dbslb"
Nov 03 23:00:00.545079 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.545055    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg4pb\" (UniqueName: \"kubernetes.io/projected/8bfb8b9d-1ad7-4c94-96ab-25f74946da5b-kube-api-access-qg4pb\") pod \"maas-api-key-cleanup-e2e-worker-w4-50d738-29895780-ckqtt\" (UID: \"8bfb8b9d-1ad7-4c94-96ab-25f74946da5b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-50d738-29895780-ckqtt"
Nov 03 23:00:00.545189 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.545083    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f6c89a6-83ae-4e3a-b10b-6da898e4857b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895780-dbslb\" (UID: \"9f6c89a6-83ae-4e3a-b10b-6da898e4857b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895780-dbslb"
Nov 03 23:00:00.545189 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.545109    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/becaf054-eec7-43bb-8443-8d5052e5859a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-5187e1-29895780-j9x65\" (UID: \"becaf054-eec7-43bb-8443-8d5052e5859a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5187e1-29895780-j9x65"
Nov 03 23:00:00.545189 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.545181    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsd5f\" (UniqueName: \"kubernetes.io/projected/1f01190d-0246-487f-a9af-54dab722b3e3-kube-api-access-zsd5f\") pod \"maas-api-key-cleanup-e2e-ipp-a-3920cb-29895780-bhr6s\" (UID: \"1f01190d-0246-487f-a9af-54dab722b3e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3920cb-29895780-bhr6s"
Nov 03 23:00:00.545330 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.545206    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxl56\" (UniqueName: \"kubernetes.io/projected/291a5765-87a6-46b2-88f4-1d4cd9056fab-kube-api-access-lxl56\") pod \"maas-api-key-cleanup-29895780-dbtlc\" (UID: \"291a5765-87a6-46b2-88f4-1d4cd9056fab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895780-dbtlc"
Nov 03 23:00:00.545500 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.545469    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdjcc\" (UniqueName: \"kubernetes.io/projected/becaf054-eec7-43bb-8443-8d5052e5859a-kube-api-access-qdjcc\") pod \"maas-api-key-cleanup-e2e-worker-w3-5187e1-29895780-j9x65\" (UID: \"becaf054-eec7-43bb-8443-8d5052e5859a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5187e1-29895780-j9x65"
Nov 03 23:00:00.545582 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.545524    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrqbx\" (UniqueName: \"kubernetes.io/projected/b6d9effb-0fd1-4443-b1ee-82600bd004ff-kube-api-access-rrqbx\") pod \"maas-api-key-cleanup-e2e-worker-w1-e19850-29895780-z748r\" (UID: \"b6d9effb-0fd1-4443-b1ee-82600bd004ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e19850-29895780-z748r"
Nov 03 23:00:00.545582 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.545564    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6d9effb-0fd1-4443-b1ee-82600bd004ff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e19850-29895780-z748r\" (UID: \"b6d9effb-0fd1-4443-b1ee-82600bd004ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e19850-29895780-z748r"
Nov 03 23:00:00.545702 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.545600    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bfb8b9d-1ad7-4c94-96ab-25f74946da5b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-50d738-29895780-ckqtt\" (UID: \"8bfb8b9d-1ad7-4c94-96ab-25f74946da5b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-50d738-29895780-ckqtt"
Nov 03 23:00:00.545702 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.545690    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04763bf4-4b53-41a0-a47d-72d46f1f9563-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-ff80c0-29895780-srjhc\" (UID: \"04763bf4-4b53-41a0-a47d-72d46f1f9563\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-ff80c0-29895780-srjhc"
Nov 03 23:00:00.545837 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.545729    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/291a5765-87a6-46b2-88f4-1d4cd9056fab-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895780-dbtlc\" (UID: \"291a5765-87a6-46b2-88f4-1d4cd9056fab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895780-dbtlc"
Nov 03 23:00:00.545837 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.545799    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f01190d-0246-487f-a9af-54dab722b3e3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-3920cb-29895780-bhr6s\" (UID: \"1f01190d-0246-487f-a9af-54dab722b3e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3920cb-29895780-bhr6s"
Nov 03 23:00:00.546001 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.545980    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f6c89a6-83ae-4e3a-b10b-6da898e4857b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895780-dbslb\" (UID: \"9f6c89a6-83ae-4e3a-b10b-6da898e4857b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895780-dbslb"
Nov 03 23:00:00.546115 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.546049    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/becaf054-eec7-43bb-8443-8d5052e5859a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-5187e1-29895780-j9x65\" (UID: \"becaf054-eec7-43bb-8443-8d5052e5859a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5187e1-29895780-j9x65"
Nov 03 23:00:00.546658 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.546636    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6d9effb-0fd1-4443-b1ee-82600bd004ff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e19850-29895780-z748r\" (UID: \"b6d9effb-0fd1-4443-b1ee-82600bd004ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e19850-29895780-z748r"
Nov 03 23:00:00.546778 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.546655    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/291a5765-87a6-46b2-88f4-1d4cd9056fab-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895780-dbtlc\" (UID: \"291a5765-87a6-46b2-88f4-1d4cd9056fab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895780-dbtlc"
Nov 03 23:00:00.546905 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.546786    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04763bf4-4b53-41a0-a47d-72d46f1f9563-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-ff80c0-29895780-srjhc\" (UID: \"04763bf4-4b53-41a0-a47d-72d46f1f9563\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-ff80c0-29895780-srjhc"
Nov 03 23:00:00.546905 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.546793    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bfb8b9d-1ad7-4c94-96ab-25f74946da5b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-50d738-29895780-ckqtt\" (UID: \"8bfb8b9d-1ad7-4c94-96ab-25f74946da5b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-50d738-29895780-ckqtt"
Nov 03 23:00:00.573051 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.570900    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qg4pb\" (UniqueName: \"kubernetes.io/projected/8bfb8b9d-1ad7-4c94-96ab-25f74946da5b-kube-api-access-qg4pb\") pod \"maas-api-key-cleanup-e2e-worker-w4-50d738-29895780-ckqtt\" (UID: \"8bfb8b9d-1ad7-4c94-96ab-25f74946da5b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-50d738-29895780-ckqtt"
Nov 03 23:00:00.577388 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.573841    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrqbx\" (UniqueName: \"kubernetes.io/projected/b6d9effb-0fd1-4443-b1ee-82600bd004ff-kube-api-access-rrqbx\") pod \"maas-api-key-cleanup-e2e-worker-w1-e19850-29895780-z748r\" (UID: \"b6d9effb-0fd1-4443-b1ee-82600bd004ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e19850-29895780-z748r"
Nov 03 23:00:00.579261 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.578291    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxl56\" (UniqueName: \"kubernetes.io/projected/291a5765-87a6-46b2-88f4-1d4cd9056fab-kube-api-access-lxl56\") pod \"maas-api-key-cleanup-29895780-dbtlc\" (UID: \"291a5765-87a6-46b2-88f4-1d4cd9056fab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895780-dbtlc"
Nov 03 23:00:00.579261 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.578802    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdjcc\" (UniqueName: \"kubernetes.io/projected/becaf054-eec7-43bb-8443-8d5052e5859a-kube-api-access-qdjcc\") pod \"maas-api-key-cleanup-e2e-worker-w3-5187e1-29895780-j9x65\" (UID: \"becaf054-eec7-43bb-8443-8d5052e5859a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5187e1-29895780-j9x65"
Nov 03 23:00:00.579261 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.578916    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsd5f\" (UniqueName: \"kubernetes.io/projected/1f01190d-0246-487f-a9af-54dab722b3e3-kube-api-access-zsd5f\") pod \"maas-api-key-cleanup-e2e-ipp-a-3920cb-29895780-bhr6s\" (UID: \"1f01190d-0246-487f-a9af-54dab722b3e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3920cb-29895780-bhr6s"
Nov 03 23:00:00.579588 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.579523    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcpgp\" (UniqueName: \"kubernetes.io/projected/04763bf4-4b53-41a0-a47d-72d46f1f9563-kube-api-access-pcpgp\") pod \"maas-api-key-cleanup-e2e-ipp-b-ff80c0-29895780-srjhc\" (UID: \"04763bf4-4b53-41a0-a47d-72d46f1f9563\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-ff80c0-29895780-srjhc"
Nov 03 23:00:00.580322 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.580300    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cm9g\" (UniqueName: \"kubernetes.io/projected/9f6c89a6-83ae-4e3a-b10b-6da898e4857b-kube-api-access-6cm9g\") pod \"maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895780-dbslb\" (UID: \"9f6c89a6-83ae-4e3a-b10b-6da898e4857b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895780-dbslb"
Nov 03 23:00:00.602307 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.601928    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-23d281-29895780-cbhhq"
Nov 03 23:00:00.652546 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.652301    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5187e1-29895780-j9x65"
Nov 03 23:00:00.668600 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.668499    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:00:00.677447 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.677172    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e19850-29895780-z748r"
Nov 03 23:00:00.684099 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.683873    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-50d738-29895780-ckqtt"
Nov 03 23:00:00.724269 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.722554    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895780-dbslb"
Nov 03 23:00:00.735768 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.735544    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895780-dbtlc"
Nov 03 23:00:00.756286 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.756100    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-ff80c0-29895780-srjhc"
Nov 03 23:00:00.811387 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.811098    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3920cb-29895780-bhr6s"
Nov 03 23:00:00.933088 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:00.932733    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-23d281-29895780-cbhhq"]
Nov 03 23:00:00.935664 ip-10-0-139-73 kubenswrapper[4431]: W1103 23:00:00.935630    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c362bb2_ed25_42d4_8907_7ea5985b3df4.slice/crio-1aa03c830ee223d5636c75c90ae8fd45fa5d78605085abb47077546ebd568962 WatchSource:0}: Error finding container 1aa03c830ee223d5636c75c90ae8fd45fa5d78605085abb47077546ebd568962: Status 404 returned error can't find the container with id 1aa03c830ee223d5636c75c90ae8fd45fa5d78605085abb47077546ebd568962
Nov 03 23:00:01.287875 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:01.287838    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5187e1-29895780-j9x65"]
Nov 03 23:00:01.297891 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:01.297861    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e19850-29895780-z748r"]
Nov 03 23:00:01.302554 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:01.302534    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-50d738-29895780-ckqtt"]
Nov 03 23:00:01.328844 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:01.328810    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895780-dbtlc"]
Nov 03 23:00:01.333063 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:01.333037    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895780-dbslb"]
Nov 03 23:00:01.371681 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:00:01.371639    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 23:00:01.402853 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:01.402823    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-ff80c0-29895780-srjhc"]
Nov 03 23:00:01.408075 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:01.408045    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3920cb-29895780-bhr6s"]
Nov 03 23:00:01.477982 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:01.477943    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-23d281-29895780-cbhhq" event={"ID":"1c362bb2-ed25-42d4-8907-7ea5985b3df4","Type":"ContainerStarted","Data":"94235ce53cef4aa8d1a0f9728b436523a8afa706acf0ac6478cf18fe04b029ce"}
Nov 03 23:00:01.478104 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:01.478043    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-23d281-29895780-cbhhq" event={"ID":"1c362bb2-ed25-42d4-8907-7ea5985b3df4","Type":"ContainerStarted","Data":"1aa03c830ee223d5636c75c90ae8fd45fa5d78605085abb47077546ebd568962"}
Nov 03 23:00:01.479537 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:01.479513    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3920cb-29895780-bhr6s" event={"ID":"1f01190d-0246-487f-a9af-54dab722b3e3","Type":"ContainerStarted","Data":"a68f9a69318fab07b6cc5a736315f2743bc476de0649f28752f5378fc3c9d467"}
Nov 03 23:00:01.480834 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:01.480812    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895780-dbslb" event={"ID":"9f6c89a6-83ae-4e3a-b10b-6da898e4857b","Type":"ContainerStarted","Data":"8398000f3c3c71f909b6fd8f29938fbdd69e29bba3cae3fa284307c2e431a88f"}
Nov 03 23:00:01.482577 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:01.482554    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-2a2eca/rate-test-model-2a2eca-kserve-7bf8dbc765-xm2vl" event={"ID":"e4e5734b-ac12-4292-beb0-77d4aa2c08d6","Type":"ContainerStarted","Data":"5cc4876f7133858a4b25b243c47556769289df93e33d0a2862f23a55b1db9847"}
Nov 03 23:00:01.482695 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:01.482587    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-2a2eca/rate-test-model-2a2eca-kserve-7bf8dbc765-xm2vl" event={"ID":"e4e5734b-ac12-4292-beb0-77d4aa2c08d6","Type":"ContainerStarted","Data":"fc0269b4eac9ca9e19ecd8955b5e2cd241328c97f5d92cb2fdd0f16579b8f132"}
Nov 03 23:00:01.483976 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:01.483949    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5187e1-29895780-j9x65" event={"ID":"becaf054-eec7-43bb-8443-8d5052e5859a","Type":"ContainerStarted","Data":"d52ac3f219092260c54c600a2060a315a9de9cb055012489ffb1675a1cc41267"}
Nov 03 23:00:01.485256 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:01.485236    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895780-dbtlc" event={"ID":"291a5765-87a6-46b2-88f4-1d4cd9056fab","Type":"ContainerStarted","Data":"ee9e88b01d35a99a56696c1aa8d6b5927f14efb3d9daf9b201aae4a800d0fbff"}
Nov 03 23:00:01.486678 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:01.486657    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-ff80c0-29895780-srjhc" event={"ID":"04763bf4-4b53-41a0-a47d-72d46f1f9563","Type":"ContainerStarted","Data":"d828cadd76775f3db5d4ea1137f0c6860e9c976618514dec6ae3ec1507f873fc"}
Nov 03 23:00:01.487914 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:01.487891    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e19850-29895780-z748r" event={"ID":"b6d9effb-0fd1-4443-b1ee-82600bd004ff","Type":"ContainerStarted","Data":"ec1be7b3f176823e9eef145b3c219afad1f596bf27c7879647186b4e92c749a8"}
Nov 03 23:00:01.489298 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:01.489276    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-50d738-29895780-ckqtt" event={"ID":"8bfb8b9d-1ad7-4c94-96ab-25f74946da5b","Type":"ContainerStarted","Data":"c474371241fe98c8424925ee542ff426174b102d03285f8ba5accb844f8dbd4c"}
Nov 03 23:00:01.522712 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:01.522666    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-23d281-29895780-cbhhq" podStartSLOduration=1.522652194 podStartE2EDuration="1.522652194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:00:01.519714794 +0000 UTC m=+2825.545570655" watchObservedRunningTime="2026-11-03 23:00:01.522652194 +0000 UTC m=+2825.548508057"
Nov 03 23:00:01.563755 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:01.563703    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w6-2a2eca/rate-test-model-2a2eca-kserve-7bf8dbc765-xm2vl" podStartSLOduration=2.563688258 podStartE2EDuration="2.563688258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:59:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:00:01.559945084 +0000 UTC m=+2825.585800946" watchObservedRunningTime="2026-11-03 23:00:01.563688258 +0000 UTC m=+2825.589544150"
Nov 03 23:00:02.091731 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:02.091700    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w6-2a2eca/rate-test-model-2a2eca-kserve-7bf8dbc765-xm2vl"
Nov 03 23:00:02.499023 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:02.498980    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5187e1-29895780-j9x65" event={"ID":"becaf054-eec7-43bb-8443-8d5052e5859a","Type":"ContainerStarted","Data":"3e3d057822a0cac55cf4c3fae1f29ff5a330f529ab1aa9201b779935ada7d8dd"}
Nov 03 23:00:02.500792 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:02.500754    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895780-dbtlc" event={"ID":"291a5765-87a6-46b2-88f4-1d4cd9056fab","Type":"ContainerStarted","Data":"1dff85cc8c8f992043407a603850deee75244b1b03544ef9ab65e85193e1cb86"}
Nov 03 23:00:02.502428 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:02.502404    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-ff80c0-29895780-srjhc" event={"ID":"04763bf4-4b53-41a0-a47d-72d46f1f9563","Type":"ContainerStarted","Data":"78f1cd3be9cd9951f5c5883c864671bb3683cac3d2ac5f71c390207d7f952872"}
Nov 03 23:00:02.504127 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:02.504096    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e19850-29895780-z748r" event={"ID":"b6d9effb-0fd1-4443-b1ee-82600bd004ff","Type":"ContainerStarted","Data":"f656efd50812e53a195a729923382beba972395ef129e3dae9bbe11602c117ea"}
Nov 03 23:00:02.505840 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:02.505816    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-50d738-29895780-ckqtt" event={"ID":"8bfb8b9d-1ad7-4c94-96ab-25f74946da5b","Type":"ContainerStarted","Data":"c138e645c9d5210f147b1f6ec465b00e45adf1fd59a59d920a3365a07defea72"}
Nov 03 23:00:02.507805 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:02.507781    4431 generic.go:356] "Generic (PLEG): container finished" podID="1c362bb2-ed25-42d4-8907-7ea5985b3df4" containerID="94235ce53cef4aa8d1a0f9728b436523a8afa706acf0ac6478cf18fe04b029ce" exitCode=60
Nov 03 23:00:02.507913 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:02.507862    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-23d281-29895780-cbhhq" event={"ID":"1c362bb2-ed25-42d4-8907-7ea5985b3df4","Type":"ContainerDied","Data":"94235ce53cef4aa8d1a0f9728b436523a8afa706acf0ac6478cf18fe04b029ce"}
Nov 03 23:00:02.508098 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:02.508082    4431 scope.go:122] "RemoveContainer" containerID="94235ce53cef4aa8d1a0f9728b436523a8afa706acf0ac6478cf18fe04b029ce"
Nov 03 23:00:02.509720 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:02.509692    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3920cb-29895780-bhr6s" event={"ID":"1f01190d-0246-487f-a9af-54dab722b3e3","Type":"ContainerStarted","Data":"d87bf2c1a9054d09ad9c90f90a0e455c1da07d37cf97c7586ab47e96908b09d7"}
Nov 03 23:00:02.511678 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:02.511646    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895780-dbslb" event={"ID":"9f6c89a6-83ae-4e3a-b10b-6da898e4857b","Type":"ContainerStarted","Data":"dec06acc886ae8a4d54a1d00fc27ebba88f8a6ff440dd1b8029cee1e2bb0e4da"}
Nov 03 23:00:02.581792 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:02.581732    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5187e1-29895780-j9x65" podStartSLOduration=2.5817130969999997 podStartE2EDuration="2.581713097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:00:02.577801478 +0000 UTC m=+2826.603657349" watchObservedRunningTime="2026-11-03 23:00:02.581713097 +0000 UTC m=+2826.607568960"
Nov 03 23:00:02.621210 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:02.621114    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e19850-29895780-z748r" podStartSLOduration=2.621095598 podStartE2EDuration="2.621095598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:00:02.620185727 +0000 UTC m=+2826.646041590" watchObservedRunningTime="2026-11-03 23:00:02.621095598 +0000 UTC m=+2826.646951462"
Nov 03 23:00:02.710533 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:02.710451    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895780-dbtlc" podStartSLOduration=2.71043284 podStartE2EDuration="2.71043284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:00:02.707661321 +0000 UTC m=+2826.733517186" watchObservedRunningTime="2026-11-03 23:00:02.71043284 +0000 UTC m=+2826.736288702"
Nov 03 23:00:02.750120 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:02.750011    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3920cb-29895780-bhr6s" podStartSLOduration=2.749990303 podStartE2EDuration="2.749990303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:00:02.746701874 +0000 UTC m=+2826.772557747" watchObservedRunningTime="2026-11-03 23:00:02.749990303 +0000 UTC m=+2826.775846166"
Nov 03 23:00:02.791025 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:02.790965    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895780-dbslb" podStartSLOduration=2.7909442110000002 podStartE2EDuration="2.790944211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:00:02.787164771 +0000 UTC m=+2826.813020633" watchObservedRunningTime="2026-11-03 23:00:02.790944211 +0000 UTC m=+2826.816800074"
Nov 03 23:00:02.811755 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:02.811713    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:00:02.834729 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:02.834667    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-ff80c0-29895780-srjhc" podStartSLOduration=2.834652374 podStartE2EDuration="2.834652374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:00:02.833721654 +0000 UTC m=+2826.859577517" watchObservedRunningTime="2026-11-03 23:00:02.834652374 +0000 UTC m=+2826.860508237"
Nov 03 23:00:02.873752 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:02.873704    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-50d738-29895780-ckqtt" podStartSLOduration=2.873688299 podStartE2EDuration="2.873688299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:00:02.872838821 +0000 UTC m=+2826.898694890" watchObservedRunningTime="2026-11-03 23:00:02.873688299 +0000 UTC m=+2826.899544162"
Nov 03 23:00:03.381911 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:03.381824    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:00:03.520897 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:03.520864    4431 generic.go:356] "Generic (PLEG): container finished" podID="1f01190d-0246-487f-a9af-54dab722b3e3" containerID="d87bf2c1a9054d09ad9c90f90a0e455c1da07d37cf97c7586ab47e96908b09d7" exitCode=60
Nov 03 23:00:03.521481 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:03.520944    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3920cb-29895780-bhr6s" event={"ID":"1f01190d-0246-487f-a9af-54dab722b3e3","Type":"ContainerDied","Data":"d87bf2c1a9054d09ad9c90f90a0e455c1da07d37cf97c7586ab47e96908b09d7"}
Nov 03 23:00:03.521481 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:03.521372    4431 scope.go:122] "RemoveContainer" containerID="d87bf2c1a9054d09ad9c90f90a0e455c1da07d37cf97c7586ab47e96908b09d7"
Nov 03 23:00:03.524903 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:03.524871    4431 generic.go:356] "Generic (PLEG): container finished" podID="9f6c89a6-83ae-4e3a-b10b-6da898e4857b" containerID="dec06acc886ae8a4d54a1d00fc27ebba88f8a6ff440dd1b8029cee1e2bb0e4da" exitCode=60
Nov 03 23:00:03.525035 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:03.524919    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895780-dbslb" event={"ID":"9f6c89a6-83ae-4e3a-b10b-6da898e4857b","Type":"ContainerDied","Data":"dec06acc886ae8a4d54a1d00fc27ebba88f8a6ff440dd1b8029cee1e2bb0e4da"}
Nov 03 23:00:03.525257 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:03.525240    4431 scope.go:122] "RemoveContainer" containerID="dec06acc886ae8a4d54a1d00fc27ebba88f8a6ff440dd1b8029cee1e2bb0e4da"
Nov 03 23:00:03.527498 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:03.527473    4431 generic.go:356] "Generic (PLEG): container finished" podID="becaf054-eec7-43bb-8443-8d5052e5859a" containerID="3e3d057822a0cac55cf4c3fae1f29ff5a330f529ab1aa9201b779935ada7d8dd" exitCode=60
Nov 03 23:00:03.527611 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:03.527559    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5187e1-29895780-j9x65" event={"ID":"becaf054-eec7-43bb-8443-8d5052e5859a","Type":"ContainerDied","Data":"3e3d057822a0cac55cf4c3fae1f29ff5a330f529ab1aa9201b779935ada7d8dd"}
Nov 03 23:00:03.527840 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:03.527821    4431 scope.go:122] "RemoveContainer" containerID="3e3d057822a0cac55cf4c3fae1f29ff5a330f529ab1aa9201b779935ada7d8dd"
Nov 03 23:00:03.529341 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:03.529319    4431 generic.go:356] "Generic (PLEG): container finished" podID="291a5765-87a6-46b2-88f4-1d4cd9056fab" containerID="1dff85cc8c8f992043407a603850deee75244b1b03544ef9ab65e85193e1cb86" exitCode=60
Nov 03 23:00:03.529446 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:03.529404    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895780-dbtlc" event={"ID":"291a5765-87a6-46b2-88f4-1d4cd9056fab","Type":"ContainerDied","Data":"1dff85cc8c8f992043407a603850deee75244b1b03544ef9ab65e85193e1cb86"}
Nov 03 23:00:03.529659 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:03.529641    4431 scope.go:122] "RemoveContainer" containerID="1dff85cc8c8f992043407a603850deee75244b1b03544ef9ab65e85193e1cb86"
Nov 03 23:00:03.531133 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:03.531112    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:00:03.531310 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:03.531287    4431 generic.go:356] "Generic (PLEG): container finished" podID="04763bf4-4b53-41a0-a47d-72d46f1f9563" containerID="78f1cd3be9cd9951f5c5883c864671bb3683cac3d2ac5f71c390207d7f952872" exitCode=60
Nov 03 23:00:03.531415 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:03.531373    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-ff80c0-29895780-srjhc" event={"ID":"04763bf4-4b53-41a0-a47d-72d46f1f9563","Type":"ContainerDied","Data":"78f1cd3be9cd9951f5c5883c864671bb3683cac3d2ac5f71c390207d7f952872"}
Nov 03 23:00:03.531630 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:03.531603    4431 scope.go:122] "RemoveContainer" containerID="78f1cd3be9cd9951f5c5883c864671bb3683cac3d2ac5f71c390207d7f952872"
Nov 03 23:00:03.536555 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:03.534262    4431 generic.go:356] "Generic (PLEG): container finished" podID="b6d9effb-0fd1-4443-b1ee-82600bd004ff" containerID="f656efd50812e53a195a729923382beba972395ef129e3dae9bbe11602c117ea" exitCode=60
Nov 03 23:00:03.536555 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:03.534370    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e19850-29895780-z748r" event={"ID":"b6d9effb-0fd1-4443-b1ee-82600bd004ff","Type":"ContainerDied","Data":"f656efd50812e53a195a729923382beba972395ef129e3dae9bbe11602c117ea"}
Nov 03 23:00:03.536555 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:03.534629    4431 scope.go:122] "RemoveContainer" containerID="f656efd50812e53a195a729923382beba972395ef129e3dae9bbe11602c117ea"
Nov 03 23:00:03.537551 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:03.537073    4431 generic.go:356] "Generic (PLEG): container finished" podID="8bfb8b9d-1ad7-4c94-96ab-25f74946da5b" containerID="c138e645c9d5210f147b1f6ec465b00e45adf1fd59a59d920a3365a07defea72" exitCode=60
Nov 03 23:00:03.537551 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:03.537336    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-50d738-29895780-ckqtt" event={"ID":"8bfb8b9d-1ad7-4c94-96ab-25f74946da5b","Type":"ContainerDied","Data":"c138e645c9d5210f147b1f6ec465b00e45adf1fd59a59d920a3365a07defea72"}
Nov 03 23:00:03.537551 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:03.537473    4431 scope.go:122] "RemoveContainer" containerID="c138e645c9d5210f147b1f6ec465b00e45adf1fd59a59d920a3365a07defea72"
Nov 03 23:00:03.543751 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:03.543724    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-23d281-29895780-cbhhq" event={"ID":"1c362bb2-ed25-42d4-8907-7ea5985b3df4","Type":"ContainerStarted","Data":"93fbc9fe96465e40d689abceb7e666d243576006b5f6a6473455dbc661f6731f"}
Nov 03 23:00:04.317086 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:04.317008    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4a01f9-6df8b8cb59-l6zlr"
Nov 03 23:00:04.317086 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:04.317076    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-4a01f9-57fb584d-fn5pv"
Nov 03 23:00:04.554011 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:04.553973    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-ff80c0-29895780-srjhc" event={"ID":"04763bf4-4b53-41a0-a47d-72d46f1f9563","Type":"ContainerStarted","Data":"5a8cb603a1e15215cd6b26d9d7dd5c15ffc09a38191a588409ce7fc3b55849b6"}
Nov 03 23:00:04.556591 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:04.556561    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e19850-29895780-z748r" event={"ID":"b6d9effb-0fd1-4443-b1ee-82600bd004ff","Type":"ContainerStarted","Data":"bf8eb16ff741d2f75e3216ce5ab74ce1739d4352f1b0369f431c173afb735d78"}
Nov 03 23:00:04.558568 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:04.558542    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-50d738-29895780-ckqtt" event={"ID":"8bfb8b9d-1ad7-4c94-96ab-25f74946da5b","Type":"ContainerStarted","Data":"63f947f6cf9d5e421bbd5a8d16c6baf7389fdfbf15faf526a5b2a9d2eda64320"}
Nov 03 23:00:04.560469 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:04.560445    4431 generic.go:356] "Generic (PLEG): container finished" podID="1c362bb2-ed25-42d4-8907-7ea5985b3df4" containerID="93fbc9fe96465e40d689abceb7e666d243576006b5f6a6473455dbc661f6731f" exitCode=60
Nov 03 23:00:04.560574 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:04.560527    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-23d281-29895780-cbhhq" event={"ID":"1c362bb2-ed25-42d4-8907-7ea5985b3df4","Type":"ContainerDied","Data":"93fbc9fe96465e40d689abceb7e666d243576006b5f6a6473455dbc661f6731f"}
Nov 03 23:00:04.560643 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:04.560581    4431 scope.go:122] "RemoveContainer" containerID="94235ce53cef4aa8d1a0f9728b436523a8afa706acf0ac6478cf18fe04b029ce"
Nov 03 23:00:04.560695 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:04.560679    4431 scope.go:122] "RemoveContainer" containerID="93fbc9fe96465e40d689abceb7e666d243576006b5f6a6473455dbc661f6731f"
Nov 03 23:00:04.560951 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:00:04.560924    4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-23d281-29895780-cbhhq_odh-ai-gateway-infra(1c362bb2-ed25-42d4-8907-7ea5985b3df4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-23d281-29895780-cbhhq" podUID="1c362bb2-ed25-42d4-8907-7ea5985b3df4"
Nov 03 23:00:04.562518 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:04.562487    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3920cb-29895780-bhr6s" event={"ID":"1f01190d-0246-487f-a9af-54dab722b3e3","Type":"ContainerStarted","Data":"0f257f005e164d175b92d4e1fe3a9bc7ace2b13830871de954474a9781f6fe48"}
Nov 03 23:00:04.564382 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:04.564359    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895780-dbslb" event={"ID":"9f6c89a6-83ae-4e3a-b10b-6da898e4857b","Type":"ContainerStarted","Data":"0d35fa8f74cbcbd50b40714b98050b657f9cd5fc8eb2e01f20caf6c422f9c2f4"}
Nov 03 23:00:04.566436 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:04.566412    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5187e1-29895780-j9x65" event={"ID":"becaf054-eec7-43bb-8443-8d5052e5859a","Type":"ContainerStarted","Data":"32f6c1b51d83c90359251121459ce7bbd50836b4a8d609b836bdff62f0930320"}
Nov 03 23:00:04.568421 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:04.568357    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895780-dbtlc" event={"ID":"291a5765-87a6-46b2-88f4-1d4cd9056fab","Type":"ContainerStarted","Data":"3ce232f9c551311b011bfbeda822f587ffdd03749dd97e0b5c0b775151756b38"}
Nov 03 23:00:04.878441 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:04.878410    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:00:05.044266 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:05.044202    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:00:05.215132 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:05.215072    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:00:05.232134 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:05.232099    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-3920cb/ipp-route-3920cb-kserve-76df875dbf-rrqjq"
Nov 03 23:00:05.250860 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:05.250814    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-3920cb/ipp-route-3920cb-kserve-76df875dbf-rrqjq"
Nov 03 23:00:05.489729 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:05.489632    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b"]
Nov 03 23:00:05.489934 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:05.489879    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b" podUID="c3553842-86f5-4f08-bc2f-4e7c83e4ca0e" containerName="istio-proxy" containerID="cri-o://c19256fa8088619af1d8330696f0349c2c24ff1f8b28dd1ba694bc54c691fe1a" gracePeriod=30
Nov 03 23:00:05.577369 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:05.577337    4431 generic.go:356] "Generic (PLEG): container finished" podID="becaf054-eec7-43bb-8443-8d5052e5859a" containerID="32f6c1b51d83c90359251121459ce7bbd50836b4a8d609b836bdff62f0930320" exitCode=60
Nov 03 23:00:05.577825 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:05.577414    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5187e1-29895780-j9x65" event={"ID":"becaf054-eec7-43bb-8443-8d5052e5859a","Type":"ContainerDied","Data":"32f6c1b51d83c90359251121459ce7bbd50836b4a8d609b836bdff62f0930320"}
Nov 03 23:00:05.577825 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:05.577482    4431 scope.go:122] "RemoveContainer" containerID="3e3d057822a0cac55cf4c3fae1f29ff5a330f529ab1aa9201b779935ada7d8dd"
Nov 03 23:00:05.577825 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:05.577703    4431 scope.go:122] "RemoveContainer" containerID="32f6c1b51d83c90359251121459ce7bbd50836b4a8d609b836bdff62f0930320"
Nov 03 23:00:05.578109 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:00:05.578085    4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-5187e1-29895780-j9x65_odh-ai-gateway-infra(becaf054-eec7-43bb-8443-8d5052e5859a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5187e1-29895780-j9x65" podUID="becaf054-eec7-43bb-8443-8d5052e5859a"
Nov 03 23:00:05.579645 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:05.579621    4431 generic.go:356] "Generic (PLEG): container finished" podID="291a5765-87a6-46b2-88f4-1d4cd9056fab" containerID="3ce232f9c551311b011bfbeda822f587ffdd03749dd97e0b5c0b775151756b38" exitCode=60
Nov 03 23:00:05.579748 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:05.579699    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895780-dbtlc" event={"ID":"291a5765-87a6-46b2-88f4-1d4cd9056fab","Type":"ContainerDied","Data":"3ce232f9c551311b011bfbeda822f587ffdd03749dd97e0b5c0b775151756b38"}
Nov 03 23:00:05.579980 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:05.579962    4431 scope.go:122] "RemoveContainer" containerID="3ce232f9c551311b011bfbeda822f587ffdd03749dd97e0b5c0b775151756b38"
Nov 03 23:00:05.580365 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:00:05.580338    4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895780-dbtlc_odh-ai-gateway-infra(291a5765-87a6-46b2-88f4-1d4cd9056fab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895780-dbtlc" podUID="291a5765-87a6-46b2-88f4-1d4cd9056fab"
Nov 03 23:00:05.581773 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:05.581748    4431 generic.go:356] "Generic (PLEG): container finished" podID="04763bf4-4b53-41a0-a47d-72d46f1f9563" containerID="5a8cb603a1e15215cd6b26d9d7dd5c15ffc09a38191a588409ce7fc3b55849b6" exitCode=60
Nov 03 23:00:05.581861 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:05.581780    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-ff80c0-29895780-srjhc" event={"ID":"04763bf4-4b53-41a0-a47d-72d46f1f9563","Type":"ContainerDied","Data":"5a8cb603a1e15215cd6b26d9d7dd5c15ffc09a38191a588409ce7fc3b55849b6"}
Nov 03 23:00:05.582042 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:05.582027    4431 scope.go:122] "RemoveContainer" containerID="5a8cb603a1e15215cd6b26d9d7dd5c15ffc09a38191a588409ce7fc3b55849b6"
Nov 03 23:00:05.582400 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:00:05.582372    4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-ff80c0-29895780-srjhc_odh-ai-gateway-infra(04763bf4-4b53-41a0-a47d-72d46f1f9563)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-ff80c0-29895780-srjhc" podUID="04763bf4-4b53-41a0-a47d-72d46f1f9563"
Nov 03 23:00:05.583783 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:05.583765    4431 generic.go:356] "Generic (PLEG): container finished" podID="b6d9effb-0fd1-4443-b1ee-82600bd004ff" containerID="bf8eb16ff741d2f75e3216ce5ab74ce1739d4352f1b0369f431c173afb735d78" exitCode=60
Nov 03 23:00:05.583879 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:05.583834    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e19850-29895780-z748r" event={"ID":"b6d9effb-0fd1-4443-b1ee-82600bd004ff","Type":"ContainerDied","Data":"bf8eb16ff741d2f75e3216ce5ab74ce1739d4352f1b0369f431c173afb735d78"}
Nov 03 23:00:05.584044 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:05.584029    4431 scope.go:122] "RemoveContainer" containerID="bf8eb16ff741d2f75e3216ce5ab74ce1739d4352f1b0369f431c173afb735d78"
Nov 03 23:00:05.584341 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:00:05.584313    4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e19850-29895780-z748r_odh-ai-gateway-infra(b6d9effb-0fd1-4443-b1ee-82600bd004ff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e19850-29895780-z748r" podUID="b6d9effb-0fd1-4443-b1ee-82600bd004ff"
Nov 03 23:00:05.585806 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:05.585784    4431 generic.go:356] "Generic (PLEG): container finished" podID="8bfb8b9d-1ad7-4c94-96ab-25f74946da5b" containerID="63f947f6cf9d5e421bbd5a8d16c6baf7389fdfbf15faf526a5b2a9d2eda64320" exitCode=60
Nov 03 23:00:05.585927 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:05.585812    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-50d738-29895780-ckqtt" event={"ID":"8bfb8b9d-1ad7-4c94-96ab-25f74946da5b","Type":"ContainerDied","Data":"63f947f6cf9d5e421bbd5a8d16c6baf7389fdfbf15faf526a5b2a9d2eda64320"}
Nov 03 23:00:05.586099 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:05.586083    4431 scope.go:122] "RemoveContainer" containerID="63f947f6cf9d5e421bbd5a8d16c6baf7389fdfbf15faf526a5b2a9d2eda64320"
Nov 03 23:00:05.586404 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:00:05.586383    4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-50d738-29895780-ckqtt_odh-ai-gateway-infra(8bfb8b9d-1ad7-4c94-96ab-25f74946da5b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-50d738-29895780-ckqtt" podUID="8bfb8b9d-1ad7-4c94-96ab-25f74946da5b"
Nov 03 23:00:05.587822 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:05.587802    4431 scope.go:122] "RemoveContainer" containerID="93fbc9fe96465e40d689abceb7e666d243576006b5f6a6473455dbc661f6731f"
Nov 03 23:00:05.588077 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:00:05.588057    4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-23d281-29895780-cbhhq_odh-ai-gateway-infra(1c362bb2-ed25-42d4-8907-7ea5985b3df4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-23d281-29895780-cbhhq" podUID="1c362bb2-ed25-42d4-8907-7ea5985b3df4"
Nov 03 23:00:05.589464 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:05.589444    4431 generic.go:356] "Generic (PLEG): container finished" podID="1f01190d-0246-487f-a9af-54dab722b3e3" containerID="0f257f005e164d175b92d4e1fe3a9bc7ace2b13830871de954474a9781f6fe48" exitCode=60
Nov 03 23:00:05.589556 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:05.589468    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3920cb-29895780-bhr6s" event={"ID":"1f01190d-0246-487f-a9af-54dab722b3e3","Type":"ContainerDied","Data":"0f257f005e164d175b92d4e1fe3a9bc7ace2b13830871de954474a9781f6fe48"}
Nov 03 23:00:05.589726 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:05.589685    4431 scope.go:122] "RemoveContainer" containerID="0f257f005e164d175b92d4e1fe3a9bc7ace2b13830871de954474a9781f6fe48"
Nov 03 23:00:05.589929 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:00:05.589909    4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-3920cb-29895780-bhr6s_odh-ai-gateway-infra(1f01190d-0246-487f-a9af-54dab722b3e3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3920cb-29895780-bhr6s" podUID="1f01190d-0246-487f-a9af-54dab722b3e3"
Nov 03 23:00:05.591378 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:05.591357    4431 generic.go:356] "Generic (PLEG): container finished" podID="9f6c89a6-83ae-4e3a-b10b-6da898e4857b" containerID="0d35fa8f74cbcbd50b40714b98050b657f9cd5fc8eb2e01f20caf6c422f9c2f4" exitCode=60
Nov 03 23:00:05.591482 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:05.591435    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895780-dbslb" event={"ID":"9f6c89a6-83ae-4e3a-b10b-6da898e4857b","Type":"ContainerDied","Data":"0d35fa8f74cbcbd50b40714b98050b657f9cd5fc8eb2e01f20caf6c422f9c2f4"}
Nov 03 23:00:05.591659 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:05.591638    4431 scope.go:122] "RemoveContainer" containerID="0d35fa8f74cbcbd50b40714b98050b657f9cd5fc8eb2e01f20caf6c422f9c2f4"
Nov 03 23:00:05.591955 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:00:05.591931    4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895780-dbslb_odh-ai-gateway-infra(9f6c89a6-83ae-4e3a-b10b-6da898e4857b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895780-dbslb" podUID="9f6c89a6-83ae-4e3a-b10b-6da898e4857b"
Nov 03 23:00:05.592291 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:05.592261    4431 scope.go:122] "RemoveContainer" containerID="1dff85cc8c8f992043407a603850deee75244b1b03544ef9ab65e85193e1cb86"
Nov 03 23:00:05.603894 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:05.603871    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-3920cb/ipp-route-3920cb-kserve-76df875dbf-rrqjq"
Nov 03 23:00:05.606249 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:05.606204    4431 scope.go:122] "RemoveContainer" containerID="78f1cd3be9cd9951f5c5883c864671bb3683cac3d2ac5f71c390207d7f952872"
Nov 03 23:00:05.622736 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:05.622714    4431 scope.go:122] "RemoveContainer" containerID="f656efd50812e53a195a729923382beba972395ef129e3dae9bbe11602c117ea"
Nov 03 23:00:05.636969 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:05.636952    4431 scope.go:122] "RemoveContainer" containerID="c138e645c9d5210f147b1f6ec465b00e45adf1fd59a59d920a3365a07defea72"
Nov 03 23:00:05.650440 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:05.650421    4431 scope.go:122] "RemoveContainer" containerID="d87bf2c1a9054d09ad9c90f90a0e455c1da07d37cf97c7586ab47e96908b09d7"
Nov 03 23:00:05.666731 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:05.666295    4431 scope.go:122] "RemoveContainer" containerID="dec06acc886ae8a4d54a1d00fc27ebba88f8a6ff440dd1b8029cee1e2bb0e4da"
Nov 03 23:00:06.139166 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:06.139128    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:00:06.255739 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:06.255629    4431 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 23:00:06.255739 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:06.255683    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b" podUID="c3553842-86f5-4f08-bc2f-4e7c83e4ca0e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 23:00:06.279836 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:06.279795    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:00:06.509335 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:06.509301    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-r4qjb"]
Nov 03 23:00:06.509572 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:06.509544    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-r4qjb" podUID="d5687c54-7c44-4806-846e-3646f096b7ab" containerName="istio-proxy" containerID="cri-o://82e28828aef5fc4f494a32d2b0c0df6bbe6b63265cb1902de1b739d7e45ba38d" gracePeriod=30
Nov 03 23:00:06.603329 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:06.603301    4431 scope.go:122] "RemoveContainer" containerID="5a8cb603a1e15215cd6b26d9d7dd5c15ffc09a38191a588409ce7fc3b55849b6"
Nov 03 23:00:06.603800 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:00:06.603715    4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-ff80c0-29895780-srjhc_odh-ai-gateway-infra(04763bf4-4b53-41a0-a47d-72d46f1f9563)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-ff80c0-29895780-srjhc" podUID="04763bf4-4b53-41a0-a47d-72d46f1f9563"
Nov 03 23:00:06.605579 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:06.605555    4431 scope.go:122] "RemoveContainer" containerID="bf8eb16ff741d2f75e3216ce5ab74ce1739d4352f1b0369f431c173afb735d78"
Nov 03 23:00:06.605994 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:00:06.605889    4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e19850-29895780-z748r_odh-ai-gateway-infra(b6d9effb-0fd1-4443-b1ee-82600bd004ff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e19850-29895780-z748r" podUID="b6d9effb-0fd1-4443-b1ee-82600bd004ff"
Nov 03 23:00:06.607775 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:06.607751    4431 scope.go:122] "RemoveContainer" containerID="63f947f6cf9d5e421bbd5a8d16c6baf7389fdfbf15faf526a5b2a9d2eda64320"
Nov 03 23:00:06.608139 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:00:06.608109    4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-50d738-29895780-ckqtt_odh-ai-gateway-infra(8bfb8b9d-1ad7-4c94-96ab-25f74946da5b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-50d738-29895780-ckqtt" podUID="8bfb8b9d-1ad7-4c94-96ab-25f74946da5b"
Nov 03 23:00:06.610171 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:06.610150    4431 scope.go:122] "RemoveContainer" containerID="0f257f005e164d175b92d4e1fe3a9bc7ace2b13830871de954474a9781f6fe48"
Nov 03 23:00:06.610502 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:00:06.610475    4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-3920cb-29895780-bhr6s_odh-ai-gateway-infra(1f01190d-0246-487f-a9af-54dab722b3e3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3920cb-29895780-bhr6s" podUID="1f01190d-0246-487f-a9af-54dab722b3e3"
Nov 03 23:00:06.612974 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:06.612751    4431 scope.go:122] "RemoveContainer" containerID="0d35fa8f74cbcbd50b40714b98050b657f9cd5fc8eb2e01f20caf6c422f9c2f4"
Nov 03 23:00:06.613084 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:00:06.613045    4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895780-dbslb_odh-ai-gateway-infra(9f6c89a6-83ae-4e3a-b10b-6da898e4857b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895780-dbslb" podUID="9f6c89a6-83ae-4e3a-b10b-6da898e4857b"
Nov 03 23:00:06.615534 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:06.615516    4431 scope.go:122] "RemoveContainer" containerID="32f6c1b51d83c90359251121459ce7bbd50836b4a8d609b836bdff62f0930320"
Nov 03 23:00:06.615878 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:00:06.615837    4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-5187e1-29895780-j9x65_odh-ai-gateway-infra(becaf054-eec7-43bb-8443-8d5052e5859a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5187e1-29895780-j9x65" podUID="becaf054-eec7-43bb-8443-8d5052e5859a"
Nov 03 23:00:06.618441 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:06.618385    4431 scope.go:122] "RemoveContainer" containerID="3ce232f9c551311b011bfbeda822f587ffdd03749dd97e0b5c0b775151756b38"
Nov 03 23:00:06.618700 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:00:06.618678    4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895780-dbtlc_odh-ai-gateway-infra(291a5765-87a6-46b2-88f4-1d4cd9056fab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895780-dbtlc" podUID="291a5765-87a6-46b2-88f4-1d4cd9056fab"
Nov 03 23:00:06.919189 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:06.919147    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d8a3d1/auth-test-model-d8a3d1-kserve-5fbc7778b9-kj9kw"]
Nov 03 23:00:06.990241 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:06.989320    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:00:07.165407 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:07.165372    4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-r4qjb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 23:00:07.165567 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:07.165434    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-r4qjb" podUID="d5687c54-7c44-4806-846e-3646f096b7ab" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 23:00:07.318058 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:07.317970    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:00:09.037759 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:09.037718    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:00:10.090903 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:10.090873    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w6-2a2eca/rate-test-model-2a2eca-kserve-7bf8dbc765-xm2vl"
Nov 03 23:00:10.098732 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:10.098675    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w6-2a2eca/rate-test-model-2a2eca-kserve-7bf8dbc765-xm2vl"
Nov 03 23:00:10.654610 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:10.654575    4431 generic.go:356] "Generic (PLEG): container finished" podID="c3553842-86f5-4f08-bc2f-4e7c83e4ca0e" containerID="c19256fa8088619af1d8330696f0349c2c24ff1f8b28dd1ba694bc54c691fe1a" exitCode=0
Nov 03 23:00:10.654805 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:10.654654    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b" event={"ID":"c3553842-86f5-4f08-bc2f-4e7c83e4ca0e","Type":"ContainerDied","Data":"c19256fa8088619af1d8330696f0349c2c24ff1f8b28dd1ba694bc54c691fe1a"}
Nov 03 23:00:10.666592 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:10.666556    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w6-2a2eca/rate-test-model-2a2eca-kserve-7bf8dbc765-xm2vl"
Nov 03 23:00:10.788879 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:10.788855    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b"
Nov 03 23:00:10.826017 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:10.825988    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:00:10.832294 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:10.832268    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3553842-86f5-4f08-bc2f-4e7c83e4ca0e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3553842-86f5-4f08-bc2f-4e7c83e4ca0e-workload-certs\") pod \"c3553842-86f5-4f08-bc2f-4e7c83e4ca0e\" (UID: \"c3553842-86f5-4f08-bc2f-4e7c83e4ca0e\") "
Nov 03 23:00:10.832403 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:10.832317    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c3553842-86f5-4f08-bc2f-4e7c83e4ca0e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3553842-86f5-4f08-bc2f-4e7c83e4ca0e-istio-podinfo\") pod \"c3553842-86f5-4f08-bc2f-4e7c83e4ca0e\" (UID: \"c3553842-86f5-4f08-bc2f-4e7c83e4ca0e\") "
Nov 03 23:00:10.832403 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:10.832380    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3553842-86f5-4f08-bc2f-4e7c83e4ca0e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3553842-86f5-4f08-bc2f-4e7c83e4ca0e-workload-socket\") pod \"c3553842-86f5-4f08-bc2f-4e7c83e4ca0e\" (UID: \"c3553842-86f5-4f08-bc2f-4e7c83e4ca0e\") "
Nov 03 23:00:10.832499 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:10.832420    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3553842-86f5-4f08-bc2f-4e7c83e4ca0e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3553842-86f5-4f08-bc2f-4e7c83e4ca0e-istio-envoy\") pod \"c3553842-86f5-4f08-bc2f-4e7c83e4ca0e\" (UID: \"c3553842-86f5-4f08-bc2f-4e7c83e4ca0e\") "
Nov 03 23:00:10.832548 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:10.832522    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3553842-86f5-4f08-bc2f-4e7c83e4ca0e-workload-certs" pod "c3553842-86f5-4f08-bc2f-4e7c83e4ca0e" (UID: "c3553842-86f5-4f08-bc2f-4e7c83e4ca0e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:00:10.832584 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:10.832555    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3553842-86f5-4f08-bc2f-4e7c83e4ca0e-istio-token\" (UniqueName: \"kubernetes.io/projected/c3553842-86f5-4f08-bc2f-4e7c83e4ca0e-istio-token\") pod \"c3553842-86f5-4f08-bc2f-4e7c83e4ca0e\" (UID: \"c3553842-86f5-4f08-bc2f-4e7c83e4ca0e\") "
Nov 03 23:00:10.832624 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:10.832600    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3553842-86f5-4f08-bc2f-4e7c83e4ca0e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3553842-86f5-4f08-bc2f-4e7c83e4ca0e-istio-data\") pod \"c3553842-86f5-4f08-bc2f-4e7c83e4ca0e\" (UID: \"c3553842-86f5-4f08-bc2f-4e7c83e4ca0e\") "
Nov 03 23:00:10.832665 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:10.832627    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3553842-86f5-4f08-bc2f-4e7c83e4ca0e-workload-socket" pod "c3553842-86f5-4f08-bc2f-4e7c83e4ca0e" (UID: "c3553842-86f5-4f08-bc2f-4e7c83e4ca0e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:00:10.832741 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:10.832717    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3553842-86f5-4f08-bc2f-4e7c83e4ca0e-kube-api-access-mw7f5\" (UniqueName: \"kubernetes.io/projected/c3553842-86f5-4f08-bc2f-4e7c83e4ca0e-kube-api-access-mw7f5\") pod \"c3553842-86f5-4f08-bc2f-4e7c83e4ca0e\" (UID: \"c3553842-86f5-4f08-bc2f-4e7c83e4ca0e\") "
Nov 03 23:00:10.832848 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:10.832828    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3553842-86f5-4f08-bc2f-4e7c83e4ca0e-istio-data" pod "c3553842-86f5-4f08-bc2f-4e7c83e4ca0e" (UID: "c3553842-86f5-4f08-bc2f-4e7c83e4ca0e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:00:10.832903 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:10.832836    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3553842-86f5-4f08-bc2f-4e7c83e4ca0e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3553842-86f5-4f08-bc2f-4e7c83e4ca0e-istiod-ca-cert\") pod \"c3553842-86f5-4f08-bc2f-4e7c83e4ca0e\" (UID: \"c3553842-86f5-4f08-bc2f-4e7c83e4ca0e\") "
Nov 03 23:00:10.832903 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:10.832891    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3553842-86f5-4f08-bc2f-4e7c83e4ca0e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3553842-86f5-4f08-bc2f-4e7c83e4ca0e-credential-socket\") pod \"c3553842-86f5-4f08-bc2f-4e7c83e4ca0e\" (UID: \"c3553842-86f5-4f08-bc2f-4e7c83e4ca0e\") "
Nov 03 23:00:10.833176 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:10.833114    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3553842-86f5-4f08-bc2f-4e7c83e4ca0e-credential-socket" pod "c3553842-86f5-4f08-bc2f-4e7c83e4ca0e" (UID: "c3553842-86f5-4f08-bc2f-4e7c83e4ca0e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:00:10.833176 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:10.833164    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3553842-86f5-4f08-bc2f-4e7c83e4ca0e-istiod-ca-cert" pod "c3553842-86f5-4f08-bc2f-4e7c83e4ca0e" (UID: "c3553842-86f5-4f08-bc2f-4e7c83e4ca0e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:00:10.833744 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:10.833675    4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3553842-86f5-4f08-bc2f-4e7c83e4ca0e-istiod-ca-cert\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:00:10.833744 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:10.833706    4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3553842-86f5-4f08-bc2f-4e7c83e4ca0e-credential-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:00:10.833744 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:10.833721    4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3553842-86f5-4f08-bc2f-4e7c83e4ca0e-workload-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:00:10.833744 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:10.833735    4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3553842-86f5-4f08-bc2f-4e7c83e4ca0e-workload-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:00:10.833744 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:10.833748    4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3553842-86f5-4f08-bc2f-4e7c83e4ca0e-istio-data\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:00:10.834999 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:10.834964    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c3553842-86f5-4f08-bc2f-4e7c83e4ca0e-istio-podinfo" pod "c3553842-86f5-4f08-bc2f-4e7c83e4ca0e" (UID: "c3553842-86f5-4f08-bc2f-4e7c83e4ca0e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 23:00:10.835103 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:10.835068    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3553842-86f5-4f08-bc2f-4e7c83e4ca0e-kube-api-access-mw7f5" pod "c3553842-86f5-4f08-bc2f-4e7c83e4ca0e" (UID: "c3553842-86f5-4f08-bc2f-4e7c83e4ca0e"). InnerVolumeSpecName "kube-api-access-mw7f5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:00:10.835257 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:10.835237    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3553842-86f5-4f08-bc2f-4e7c83e4ca0e-istio-envoy" pod "c3553842-86f5-4f08-bc2f-4e7c83e4ca0e" (UID: "c3553842-86f5-4f08-bc2f-4e7c83e4ca0e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:00:10.836484 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:10.836452    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3553842-86f5-4f08-bc2f-4e7c83e4ca0e-istio-token" pod "c3553842-86f5-4f08-bc2f-4e7c83e4ca0e" (UID: "c3553842-86f5-4f08-bc2f-4e7c83e4ca0e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:00:10.935597 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:10.935564    4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3553842-86f5-4f08-bc2f-4e7c83e4ca0e-istio-podinfo\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:00:10.935597 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:10.935595    4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3553842-86f5-4f08-bc2f-4e7c83e4ca0e-istio-envoy\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:00:10.935597 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:10.935607    4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3553842-86f5-4f08-bc2f-4e7c83e4ca0e-istio-token\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:00:10.935844 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:10.935619    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mw7f5\" (UniqueName: \"kubernetes.io/projected/c3553842-86f5-4f08-bc2f-4e7c83e4ca0e-kube-api-access-mw7f5\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:00:11.665512 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:11.665475    4431 generic.go:356] "Generic (PLEG): container finished" podID="d5687c54-7c44-4806-846e-3646f096b7ab" containerID="82e28828aef5fc4f494a32d2b0c0df6bbe6b63265cb1902de1b739d7e45ba38d" exitCode=0
Nov 03 23:00:11.665969 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:11.665551    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-r4qjb" event={"ID":"d5687c54-7c44-4806-846e-3646f096b7ab","Type":"ContainerDied","Data":"82e28828aef5fc4f494a32d2b0c0df6bbe6b63265cb1902de1b739d7e45ba38d"}
Nov 03 23:00:11.667454 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:11.667424    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b"
Nov 03 23:00:11.667595 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:11.667449    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b" event={"ID":"c3553842-86f5-4f08-bc2f-4e7c83e4ca0e","Type":"ContainerDied","Data":"a02b299bc47a72d3f801ec620a8ef9b98425160c522c3a983501c1c53313b960"}
Nov 03 23:00:11.667595 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:11.667503    4431 scope.go:122] "RemoveContainer" containerID="c19256fa8088619af1d8330696f0349c2c24ff1f8b28dd1ba694bc54c691fe1a"
Nov 03 23:00:11.734305 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:11.734268    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b"]
Nov 03 23:00:11.746773 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:11.746739    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-shz2b"]
Nov 03 23:00:11.801129 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:11.801104    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-r4qjb"
Nov 03 23:00:11.848628 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:11.848587    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d5687c54-7c44-4806-846e-3646f096b7ab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5687c54-7c44-4806-846e-3646f096b7ab-istio-podinfo\") pod \"d5687c54-7c44-4806-846e-3646f096b7ab\" (UID: \"d5687c54-7c44-4806-846e-3646f096b7ab\") "
Nov 03 23:00:11.848846 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:11.848753    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5687c54-7c44-4806-846e-3646f096b7ab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5687c54-7c44-4806-846e-3646f096b7ab-istiod-ca-cert\") pod \"d5687c54-7c44-4806-846e-3646f096b7ab\" (UID: \"d5687c54-7c44-4806-846e-3646f096b7ab\") "
Nov 03 23:00:11.848846 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:11.848814    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5687c54-7c44-4806-846e-3646f096b7ab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5687c54-7c44-4806-846e-3646f096b7ab-workload-certs\") pod \"d5687c54-7c44-4806-846e-3646f096b7ab\" (UID: \"d5687c54-7c44-4806-846e-3646f096b7ab\") "
Nov 03 23:00:11.848981 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:11.848868    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5687c54-7c44-4806-846e-3646f096b7ab-istio-token\" (UniqueName: \"kubernetes.io/projected/d5687c54-7c44-4806-846e-3646f096b7ab-istio-token\") pod \"d5687c54-7c44-4806-846e-3646f096b7ab\" (UID: \"d5687c54-7c44-4806-846e-3646f096b7ab\") "
Nov 03 23:00:11.849022 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:11.849006    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5687c54-7c44-4806-846e-3646f096b7ab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5687c54-7c44-4806-846e-3646f096b7ab-istio-envoy\") pod \"d5687c54-7c44-4806-846e-3646f096b7ab\" (UID: \"d5687c54-7c44-4806-846e-3646f096b7ab\") "
Nov 03 23:00:11.849067 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:11.849040    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5687c54-7c44-4806-846e-3646f096b7ab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5687c54-7c44-4806-846e-3646f096b7ab-credential-socket\") pod \"d5687c54-7c44-4806-846e-3646f096b7ab\" (UID: \"d5687c54-7c44-4806-846e-3646f096b7ab\") "
Nov 03 23:00:11.849110 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:11.849076    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5687c54-7c44-4806-846e-3646f096b7ab-kube-api-access-c8xff\" (UniqueName: \"kubernetes.io/projected/d5687c54-7c44-4806-846e-3646f096b7ab-kube-api-access-c8xff\") pod \"d5687c54-7c44-4806-846e-3646f096b7ab\" (UID: \"d5687c54-7c44-4806-846e-3646f096b7ab\") "
Nov 03 23:00:11.849161 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:11.849136    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5687c54-7c44-4806-846e-3646f096b7ab-workload-certs" pod "d5687c54-7c44-4806-846e-3646f096b7ab" (UID: "d5687c54-7c44-4806-846e-3646f096b7ab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:00:11.849231 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:11.849174    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5687c54-7c44-4806-846e-3646f096b7ab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5687c54-7c44-4806-846e-3646f096b7ab-istio-data\") pod \"d5687c54-7c44-4806-846e-3646f096b7ab\" (UID: \"d5687c54-7c44-4806-846e-3646f096b7ab\") "
Nov 03 23:00:11.849277 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:11.849229    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5687c54-7c44-4806-846e-3646f096b7ab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5687c54-7c44-4806-846e-3646f096b7ab-workload-socket\") pod \"d5687c54-7c44-4806-846e-3646f096b7ab\" (UID: \"d5687c54-7c44-4806-846e-3646f096b7ab\") "
Nov 03 23:00:11.849873 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:11.849849    4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5687c54-7c44-4806-846e-3646f096b7ab-workload-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:00:11.850108 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:11.850087    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5687c54-7c44-4806-846e-3646f096b7ab-workload-socket" pod "d5687c54-7c44-4806-846e-3646f096b7ab" (UID: "d5687c54-7c44-4806-846e-3646f096b7ab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:00:11.850517 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:11.850477    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5687c54-7c44-4806-846e-3646f096b7ab-istio-data" pod "d5687c54-7c44-4806-846e-3646f096b7ab" (UID: "d5687c54-7c44-4806-846e-3646f096b7ab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:00:11.850517 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:11.850478    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5687c54-7c44-4806-846e-3646f096b7ab-istiod-ca-cert" pod "d5687c54-7c44-4806-846e-3646f096b7ab" (UID: "d5687c54-7c44-4806-846e-3646f096b7ab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:00:11.850724 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:11.850704    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5687c54-7c44-4806-846e-3646f096b7ab-credential-socket" pod "d5687c54-7c44-4806-846e-3646f096b7ab" (UID: "d5687c54-7c44-4806-846e-3646f096b7ab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:00:11.852355 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:11.852336    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d5687c54-7c44-4806-846e-3646f096b7ab-istio-podinfo" pod "d5687c54-7c44-4806-846e-3646f096b7ab" (UID: "d5687c54-7c44-4806-846e-3646f096b7ab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 23:00:11.852497 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:11.852348    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5687c54-7c44-4806-846e-3646f096b7ab-istio-envoy" pod "d5687c54-7c44-4806-846e-3646f096b7ab" (UID: "d5687c54-7c44-4806-846e-3646f096b7ab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:00:11.852560 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:11.852503    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5687c54-7c44-4806-846e-3646f096b7ab-istio-token" pod "d5687c54-7c44-4806-846e-3646f096b7ab" (UID: "d5687c54-7c44-4806-846e-3646f096b7ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:00:11.852756 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:11.852723    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5687c54-7c44-4806-846e-3646f096b7ab-kube-api-access-c8xff" pod "d5687c54-7c44-4806-846e-3646f096b7ab" (UID: "d5687c54-7c44-4806-846e-3646f096b7ab"). InnerVolumeSpecName "kube-api-access-c8xff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:00:11.951120 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:11.951086    4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5687c54-7c44-4806-846e-3646f096b7ab-istio-token\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:00:11.951120 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:11.951115    4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5687c54-7c44-4806-846e-3646f096b7ab-istio-envoy\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:00:11.951120 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:11.951124    4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5687c54-7c44-4806-846e-3646f096b7ab-credential-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:00:11.951395 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:11.951134    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8xff\" (UniqueName: \"kubernetes.io/projected/d5687c54-7c44-4806-846e-3646f096b7ab-kube-api-access-c8xff\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:00:11.951395 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:11.951143    4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5687c54-7c44-4806-846e-3646f096b7ab-istio-data\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:00:11.951395 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:11.951152    4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5687c54-7c44-4806-846e-3646f096b7ab-workload-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:00:11.951395 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:11.951160    4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5687c54-7c44-4806-846e-3646f096b7ab-istio-podinfo\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:00:11.951395 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:11.951169    4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5687c54-7c44-4806-846e-3646f096b7ab-istiod-ca-cert\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:00:12.089822 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:12.089745    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3553842-86f5-4f08-bc2f-4e7c83e4ca0e" path="/var/lib/kubelet/pods/c3553842-86f5-4f08-bc2f-4e7c83e4ca0e/volumes"
Nov 03 23:00:12.449119 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:12.449086    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:00:12.675946 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:12.675918    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-r4qjb"
Nov 03 23:00:12.676439 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:12.675954    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-r4qjb" event={"ID":"d5687c54-7c44-4806-846e-3646f096b7ab","Type":"ContainerDied","Data":"6a92df69fa358206bac87ae467cea04492ccc5315b0a1e398d8104d3abc36037"}
Nov 03 23:00:12.676439 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:12.676013    4431 scope.go:122] "RemoveContainer" containerID="82e28828aef5fc4f494a32d2b0c0df6bbe6b63265cb1902de1b739d7e45ba38d"
Nov 03 23:00:12.741325 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:12.741242    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-r4qjb"]
Nov 03 23:00:12.753229 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:12.753185    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-r4qjb"]
Nov 03 23:00:14.087312 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:14.087280    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:00:14.090053 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:14.090028    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5687c54-7c44-4806-846e-3646f096b7ab" path="/var/lib/kubelet/pods/d5687c54-7c44-4806-846e-3646f096b7ab/volumes"
Nov 03 23:00:15.080899 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:15.080866    4431 scope.go:122] "RemoveContainer" containerID="63f947f6cf9d5e421bbd5a8d16c6baf7389fdfbf15faf526a5b2a9d2eda64320"
Nov 03 23:00:15.706694 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:15.706659    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-50d738-29895780-ckqtt" event={"ID":"8bfb8b9d-1ad7-4c94-96ab-25f74946da5b","Type":"ContainerStarted","Data":"f131a63c9547e8460960d09ff51c92778ffc43be830ad6e8d4828df38e61972c"}
Nov 03 23:00:15.816445 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:15.816409    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm"]
Nov 03 23:00:15.817309 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:15.817275    4431 cpu_manager.go:414] "containerMap: removing container" podUID="c3553842-86f5-4f08-bc2f-4e7c83e4ca0e" containerName="istio-proxy"
Nov 03 23:00:15.817309 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:15.817308    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3553842-86f5-4f08-bc2f-4e7c83e4ca0e" containerName="istio-proxy"
Nov 03 23:00:15.817309 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:15.817319    4431 cpu_manager.go:414] "containerMap: removing container" podUID="d5687c54-7c44-4806-846e-3646f096b7ab" containerName="istio-proxy"
Nov 03 23:00:15.817505 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:15.817324    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5687c54-7c44-4806-846e-3646f096b7ab" containerName="istio-proxy"
Nov 03 23:00:15.817505 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:15.817480    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3553842-86f5-4f08-bc2f-4e7c83e4ca0e" containerName="istio-proxy"
Nov 03 23:00:15.817505 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:15.817488    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5687c54-7c44-4806-846e-3646f096b7ab" containerName="istio-proxy"
Nov 03 23:00:15.821553 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:15.821534    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm"
Nov 03 23:00:15.839864 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:15.839841    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-a940d5-openshift-default-dockercfg-9b7lq\""
Nov 03 23:00:15.845028 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:15.844994    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm"]
Nov 03 23:00:15.883050 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:15.883018    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:00:15.906077 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:15.906041    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44002d96-03c6-4063-b992-a531d4fe8016-workload-certs\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm\" (UID: \"44002d96-03c6-4063-b992-a531d4fe8016\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm"
Nov 03 23:00:15.906290 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:15.906119    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-492tn\" (UniqueName: \"kubernetes.io/projected/44002d96-03c6-4063-b992-a531d4fe8016-kube-api-access-492tn\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm\" (UID: \"44002d96-03c6-4063-b992-a531d4fe8016\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm"
Nov 03 23:00:15.906290 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:15.906183    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44002d96-03c6-4063-b992-a531d4fe8016-istiod-ca-cert\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm\" (UID: \"44002d96-03c6-4063-b992-a531d4fe8016\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm"
Nov 03 23:00:15.906290 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:15.906256    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44002d96-03c6-4063-b992-a531d4fe8016-istio-data\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm\" (UID: \"44002d96-03c6-4063-b992-a531d4fe8016\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm"
Nov 03 23:00:15.906447 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:15.906300    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44002d96-03c6-4063-b992-a531d4fe8016-istio-envoy\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm\" (UID: \"44002d96-03c6-4063-b992-a531d4fe8016\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm"
Nov 03 23:00:15.906495 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:15.906434    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44002d96-03c6-4063-b992-a531d4fe8016-istio-token\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm\" (UID: \"44002d96-03c6-4063-b992-a531d4fe8016\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm"
Nov 03 23:00:15.906495 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:15.906488    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44002d96-03c6-4063-b992-a531d4fe8016-istio-podinfo\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm\" (UID: \"44002d96-03c6-4063-b992-a531d4fe8016\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm"
Nov 03 23:00:15.906660 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:15.906636    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44002d96-03c6-4063-b992-a531d4fe8016-workload-socket\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm\" (UID: \"44002d96-03c6-4063-b992-a531d4fe8016\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm"
Nov 03 23:00:15.906726 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:15.906673    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44002d96-03c6-4063-b992-a531d4fe8016-credential-socket\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm\" (UID: \"44002d96-03c6-4063-b992-a531d4fe8016\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm"
Nov 03 23:00:16.008113 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:16.008036    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44002d96-03c6-4063-b992-a531d4fe8016-istio-podinfo\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm\" (UID: \"44002d96-03c6-4063-b992-a531d4fe8016\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm"
Nov 03 23:00:16.008295 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:16.008148    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44002d96-03c6-4063-b992-a531d4fe8016-workload-socket\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm\" (UID: \"44002d96-03c6-4063-b992-a531d4fe8016\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm"
Nov 03 23:00:16.008295 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:16.008173    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44002d96-03c6-4063-b992-a531d4fe8016-credential-socket\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm\" (UID: \"44002d96-03c6-4063-b992-a531d4fe8016\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm"
Nov 03 23:00:16.008402 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:16.008340    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44002d96-03c6-4063-b992-a531d4fe8016-workload-certs\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm\" (UID: \"44002d96-03c6-4063-b992-a531d4fe8016\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm"
Nov 03 23:00:16.008402 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:16.008375    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-492tn\" (UniqueName: \"kubernetes.io/projected/44002d96-03c6-4063-b992-a531d4fe8016-kube-api-access-492tn\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm\" (UID: \"44002d96-03c6-4063-b992-a531d4fe8016\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm"
Nov 03 23:00:16.008588 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:16.008423    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44002d96-03c6-4063-b992-a531d4fe8016-istiod-ca-cert\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm\" (UID: \"44002d96-03c6-4063-b992-a531d4fe8016\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm"
Nov 03 23:00:16.008588 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:16.008486    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44002d96-03c6-4063-b992-a531d4fe8016-istio-data\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm\" (UID: \"44002d96-03c6-4063-b992-a531d4fe8016\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm"
Nov 03 23:00:16.008588 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:16.008545    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44002d96-03c6-4063-b992-a531d4fe8016-istio-envoy\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm\" (UID: \"44002d96-03c6-4063-b992-a531d4fe8016\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm"
Nov 03 23:00:16.008588 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:16.008585    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44002d96-03c6-4063-b992-a531d4fe8016-istio-token\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm\" (UID: \"44002d96-03c6-4063-b992-a531d4fe8016\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm"
Nov 03 23:00:16.008833 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:16.008645    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44002d96-03c6-4063-b992-a531d4fe8016-credential-socket\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm\" (UID: \"44002d96-03c6-4063-b992-a531d4fe8016\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm"
Nov 03 23:00:16.008833 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:16.008722    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44002d96-03c6-4063-b992-a531d4fe8016-workload-certs\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm\" (UID: \"44002d96-03c6-4063-b992-a531d4fe8016\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm"
Nov 03 23:00:16.008833 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:16.008791    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44002d96-03c6-4063-b992-a531d4fe8016-workload-socket\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm\" (UID: \"44002d96-03c6-4063-b992-a531d4fe8016\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm"
Nov 03 23:00:16.009122 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:16.009095    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44002d96-03c6-4063-b992-a531d4fe8016-istio-data\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm\" (UID: \"44002d96-03c6-4063-b992-a531d4fe8016\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm"
Nov 03 23:00:16.009370 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:16.009353    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44002d96-03c6-4063-b992-a531d4fe8016-istiod-ca-cert\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm\" (UID: \"44002d96-03c6-4063-b992-a531d4fe8016\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm"
Nov 03 23:00:16.011142 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:16.011122    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44002d96-03c6-4063-b992-a531d4fe8016-istio-envoy\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm\" (UID: \"44002d96-03c6-4063-b992-a531d4fe8016\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm"
Nov 03 23:00:16.011293 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:16.011276    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44002d96-03c6-4063-b992-a531d4fe8016-istio-podinfo\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm\" (UID: \"44002d96-03c6-4063-b992-a531d4fe8016\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm"
Nov 03 23:00:16.031331 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:16.031300    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44002d96-03c6-4063-b992-a531d4fe8016-istio-token\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm\" (UID: \"44002d96-03c6-4063-b992-a531d4fe8016\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm"
Nov 03 23:00:16.031438 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:16.031365    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-492tn\" (UniqueName: \"kubernetes.io/projected/44002d96-03c6-4063-b992-a531d4fe8016-kube-api-access-492tn\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm\" (UID: \"44002d96-03c6-4063-b992-a531d4fe8016\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm"
Nov 03 23:00:16.136115 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:16.136081    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm"
Nov 03 23:00:16.458496 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:16.458469    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm"]
Nov 03 23:00:16.459304 ip-10-0-139-73 kubenswrapper[4431]: W1103 23:00:16.459271    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44002d96_03c6_4063_b992_a531d4fe8016.slice/crio-c8baf491511f5b7a97410689fea32f466877f713feb51532ef78f7ecedbdb7b4 WatchSource:0}: Error finding container c8baf491511f5b7a97410689fea32f466877f713feb51532ef78f7ecedbdb7b4: Status 404 returned error can't find the container with id c8baf491511f5b7a97410689fea32f466877f713feb51532ef78f7ecedbdb7b4
Nov 03 23:00:16.462315 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:16.462278    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:00:16.462402 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:16.462343    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:00:16.462402 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:16.462372    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:00:16.716351 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:16.716194    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm" event={"ID":"44002d96-03c6-4063-b992-a531d4fe8016","Type":"ContainerStarted","Data":"19c94ca123cc93fdc602f466ad17a5424546cee8395eec548c6d2c4a216cd062"}
Nov 03 23:00:16.716351 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:16.716284    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm" event={"ID":"44002d96-03c6-4063-b992-a531d4fe8016","Type":"ContainerStarted","Data":"c8baf491511f5b7a97410689fea32f466877f713feb51532ef78f7ecedbdb7b4"}
Nov 03 23:00:16.718652 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:16.718616    4431 generic.go:356] "Generic (PLEG): container finished" podID="8bfb8b9d-1ad7-4c94-96ab-25f74946da5b" containerID="f131a63c9547e8460960d09ff51c92778ffc43be830ad6e8d4828df38e61972c" exitCode=60
Nov 03 23:00:16.718808 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:16.718676    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-50d738-29895780-ckqtt" event={"ID":"8bfb8b9d-1ad7-4c94-96ab-25f74946da5b","Type":"ContainerDied","Data":"f131a63c9547e8460960d09ff51c92778ffc43be830ad6e8d4828df38e61972c"}
Nov 03 23:00:16.718808 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:16.718719    4431 scope.go:122] "RemoveContainer" containerID="63f947f6cf9d5e421bbd5a8d16c6baf7389fdfbf15faf526a5b2a9d2eda64320"
Nov 03 23:00:16.719091 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:16.719071    4431 scope.go:122] "RemoveContainer" containerID="f131a63c9547e8460960d09ff51c92778ffc43be830ad6e8d4828df38e61972c"
Nov 03 23:00:16.719458 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:00:16.719434    4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-50d738-29895780-ckqtt_odh-ai-gateway-infra(8bfb8b9d-1ad7-4c94-96ab-25f74946da5b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-50d738-29895780-ckqtt" podUID="8bfb8b9d-1ad7-4c94-96ab-25f74946da5b"
Nov 03 23:00:16.760494 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:16.760243    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm" podStartSLOduration=1.760225127 podStartE2EDuration="1.760225127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:00:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:00:16.759490844 +0000 UTC m=+2840.785346708" watchObservedRunningTime="2026-11-03 23:00:16.760225127 +0000 UTC m=+2840.786080982"
Nov 03 23:00:16.775654 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:16.775612    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-50d738-29895780-ckqtt"]
Nov 03 23:00:17.080774 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:17.080688    4431 scope.go:122] "RemoveContainer" containerID="3ce232f9c551311b011bfbeda822f587ffdd03749dd97e0b5c0b775151756b38"
Nov 03 23:00:17.137801 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:17.137768    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm"
Nov 03 23:00:17.140057 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:17.140026    4431 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Nov 03 23:00:17.140298 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:17.140272    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm" podUID="44002d96-03c6-4063-b992-a531d4fe8016" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Nov 03 23:00:17.730439 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:17.730347    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895780-dbtlc" event={"ID":"291a5765-87a6-46b2-88f4-1d4cd9056fab","Type":"ContainerStarted","Data":"5ccbadaf1a38a60332fd44fb37c0dad71ece2bd253965fa627e120686d03d93f"}
Nov 03 23:00:17.987696 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:17.987639    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-50d738-29895780-ckqtt"
Nov 03 23:00:18.056237 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:18.048893    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bfb8b9d-1ad7-4c94-96ab-25f74946da5b-kube-api-access-qg4pb\" (UniqueName: \"kubernetes.io/projected/8bfb8b9d-1ad7-4c94-96ab-25f74946da5b-kube-api-access-qg4pb\") pod \"8bfb8b9d-1ad7-4c94-96ab-25f74946da5b\" (UID: \"8bfb8b9d-1ad7-4c94-96ab-25f74946da5b\") "
Nov 03 23:00:18.056237 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:18.048989    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bfb8b9d-1ad7-4c94-96ab-25f74946da5b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bfb8b9d-1ad7-4c94-96ab-25f74946da5b-maas-api-service-ca\") pod \"8bfb8b9d-1ad7-4c94-96ab-25f74946da5b\" (UID: \"8bfb8b9d-1ad7-4c94-96ab-25f74946da5b\") "
Nov 03 23:00:18.056237 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:18.050338    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bfb8b9d-1ad7-4c94-96ab-25f74946da5b-maas-api-service-ca" pod "8bfb8b9d-1ad7-4c94-96ab-25f74946da5b" (UID: "8bfb8b9d-1ad7-4c94-96ab-25f74946da5b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:00:18.081237 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:18.057836    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bfb8b9d-1ad7-4c94-96ab-25f74946da5b-kube-api-access-qg4pb" pod "8bfb8b9d-1ad7-4c94-96ab-25f74946da5b" (UID: "8bfb8b9d-1ad7-4c94-96ab-25f74946da5b"). InnerVolumeSpecName "kube-api-access-qg4pb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:00:18.085258 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:18.084905    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:00:18.085737 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:18.085719    4431 scope.go:122] "RemoveContainer" containerID="bf8eb16ff741d2f75e3216ce5ab74ce1739d4352f1b0369f431c173afb735d78"
Nov 03 23:00:18.085990 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:18.085974    4431 scope.go:122] "RemoveContainer" containerID="93fbc9fe96465e40d689abceb7e666d243576006b5f6a6473455dbc661f6731f"
Nov 03 23:00:18.086180 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:18.086148    4431 scope.go:122] "RemoveContainer" containerID="32f6c1b51d83c90359251121459ce7bbd50836b4a8d609b836bdff62f0930320"
Nov 03 23:00:18.137824 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:18.137791    4431 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Nov 03 23:00:18.137991 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:18.137847    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm" podUID="44002d96-03c6-4063-b992-a531d4fe8016" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Nov 03 23:00:18.163242 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:18.158942    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qg4pb\" (UniqueName: \"kubernetes.io/projected/8bfb8b9d-1ad7-4c94-96ab-25f74946da5b-kube-api-access-qg4pb\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:00:18.163242 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:18.158973    4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bfb8b9d-1ad7-4c94-96ab-25f74946da5b-maas-api-service-ca\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:00:18.771428 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:18.771386    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-50d738-29895780-ckqtt" event={"ID":"8bfb8b9d-1ad7-4c94-96ab-25f74946da5b","Type":"ContainerDied","Data":"c474371241fe98c8424925ee542ff426174b102d03285f8ba5accb844f8dbd4c"}
Nov 03 23:00:18.771922 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:18.771463    4431 scope.go:122] "RemoveContainer" containerID="f131a63c9547e8460960d09ff51c92778ffc43be830ad6e8d4828df38e61972c"
Nov 03 23:00:18.771922 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:18.771505    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-50d738-29895780-ckqtt"
Nov 03 23:00:18.775970 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:18.775949    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-23d281-29895780-cbhhq" event={"ID":"1c362bb2-ed25-42d4-8907-7ea5985b3df4","Type":"ContainerStarted","Data":"91cdc4fcab5fca1ef64e04abe1c37f7e6e23e22c95bf2a27e2d9dbe28893ea78"}
Nov 03 23:00:18.778412 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:18.778367    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5187e1-29895780-j9x65" event={"ID":"becaf054-eec7-43bb-8443-8d5052e5859a","Type":"ContainerStarted","Data":"dc134a7b64f06237e11feec6bfd17b48f0775c0e79bd55f26c9cc9a3a1817c39"}
Nov 03 23:00:18.780950 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:18.780925    4431 generic.go:356] "Generic (PLEG): container finished" podID="291a5765-87a6-46b2-88f4-1d4cd9056fab" containerID="5ccbadaf1a38a60332fd44fb37c0dad71ece2bd253965fa627e120686d03d93f" exitCode=60
Nov 03 23:00:18.781053 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:18.781007    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895780-dbtlc" event={"ID":"291a5765-87a6-46b2-88f4-1d4cd9056fab","Type":"ContainerDied","Data":"5ccbadaf1a38a60332fd44fb37c0dad71ece2bd253965fa627e120686d03d93f"}
Nov 03 23:00:18.782816 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:18.782434    4431 scope.go:122] "RemoveContainer" containerID="5ccbadaf1a38a60332fd44fb37c0dad71ece2bd253965fa627e120686d03d93f"
Nov 03 23:00:18.782816 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:00:18.782672    4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895780-dbtlc_odh-ai-gateway-infra(291a5765-87a6-46b2-88f4-1d4cd9056fab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895780-dbtlc" podUID="291a5765-87a6-46b2-88f4-1d4cd9056fab"
Nov 03 23:00:18.786158 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:18.785636    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e19850-29895780-z748r" event={"ID":"b6d9effb-0fd1-4443-b1ee-82600bd004ff","Type":"ContainerStarted","Data":"e23e406f8f5b63623666f81d3bdc5f1dc561e2f225c85a8ef21340b801dbdcb9"}
Nov 03 23:00:18.787141 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:18.787109    4431 scope.go:122] "RemoveContainer" containerID="3ce232f9c551311b011bfbeda822f587ffdd03749dd97e0b5c0b775151756b38"
Nov 03 23:00:18.788394 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:18.788052    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895780-dbtlc"]
Nov 03 23:00:18.926745 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:18.926712    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-50d738-29895780-ckqtt"]
Nov 03 23:00:18.931248 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:18.931198    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-50d738-29895780-ckqtt"]
Nov 03 23:00:19.080832 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:19.080799    4431 scope.go:122] "RemoveContainer" containerID="0f257f005e164d175b92d4e1fe3a9bc7ace2b13830871de954474a9781f6fe48"
Nov 03 23:00:19.081004 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:19.080921    4431 scope.go:122] "RemoveContainer" containerID="5a8cb603a1e15215cd6b26d9d7dd5c15ffc09a38191a588409ce7fc3b55849b6"
Nov 03 23:00:19.081070 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:19.081051    4431 scope.go:122] "RemoveContainer" containerID="0d35fa8f74cbcbd50b40714b98050b657f9cd5fc8eb2e01f20caf6c422f9c2f4"
Nov 03 23:00:19.137337 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:19.137238    4431 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Nov 03 23:00:19.137337 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:19.137290    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm" podUID="44002d96-03c6-4063-b992-a531d4fe8016" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Nov 03 23:00:19.793559 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:19.793528    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-ff80c0-29895780-srjhc" event={"ID":"04763bf4-4b53-41a0-a47d-72d46f1f9563","Type":"ContainerStarted","Data":"b4f789269b1f142b238e9b9ac4e1a37c3966b1f0f4adb332d7533db473b45f5b"}
Nov 03 23:00:19.795862 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:19.795838    4431 generic.go:356] "Generic (PLEG): container finished" podID="b6d9effb-0fd1-4443-b1ee-82600bd004ff" containerID="e23e406f8f5b63623666f81d3bdc5f1dc561e2f225c85a8ef21340b801dbdcb9" exitCode=60
Nov 03 23:00:19.795995 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:19.795915    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e19850-29895780-z748r" event={"ID":"b6d9effb-0fd1-4443-b1ee-82600bd004ff","Type":"ContainerDied","Data":"e23e406f8f5b63623666f81d3bdc5f1dc561e2f225c85a8ef21340b801dbdcb9"}
Nov 03 23:00:19.795995 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:19.795978    4431 scope.go:122] "RemoveContainer" containerID="bf8eb16ff741d2f75e3216ce5ab74ce1739d4352f1b0369f431c173afb735d78"
Nov 03 23:00:19.796376 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:19.796357    4431 scope.go:122] "RemoveContainer" containerID="e23e406f8f5b63623666f81d3bdc5f1dc561e2f225c85a8ef21340b801dbdcb9"
Nov 03 23:00:19.796627 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:00:19.796610    4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e19850-29895780-z748r_odh-ai-gateway-infra(b6d9effb-0fd1-4443-b1ee-82600bd004ff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e19850-29895780-z748r" podUID="b6d9effb-0fd1-4443-b1ee-82600bd004ff"
Nov 03 23:00:19.799557 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:19.799537    4431 generic.go:356] "Generic (PLEG): container finished" podID="1c362bb2-ed25-42d4-8907-7ea5985b3df4" containerID="91cdc4fcab5fca1ef64e04abe1c37f7e6e23e22c95bf2a27e2d9dbe28893ea78" exitCode=60
Nov 03 23:00:19.799659 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:19.799603    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-23d281-29895780-cbhhq" event={"ID":"1c362bb2-ed25-42d4-8907-7ea5985b3df4","Type":"ContainerDied","Data":"91cdc4fcab5fca1ef64e04abe1c37f7e6e23e22c95bf2a27e2d9dbe28893ea78"}
Nov 03 23:00:19.799955 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:19.799937    4431 scope.go:122] "RemoveContainer" containerID="91cdc4fcab5fca1ef64e04abe1c37f7e6e23e22c95bf2a27e2d9dbe28893ea78"
Nov 03 23:00:19.800258 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:00:19.800233    4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-23d281-29895780-cbhhq_odh-ai-gateway-infra(1c362bb2-ed25-42d4-8907-7ea5985b3df4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-23d281-29895780-cbhhq" podUID="1c362bb2-ed25-42d4-8907-7ea5985b3df4"
Nov 03 23:00:19.801823 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:19.801803    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3920cb-29895780-bhr6s" event={"ID":"1f01190d-0246-487f-a9af-54dab722b3e3","Type":"ContainerStarted","Data":"601876aadd7f8ddaa8440c65a46fef521560a6648bc576ddea9026690f2ababe"}
Nov 03 23:00:19.803914 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:19.803893    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895780-dbslb" event={"ID":"9f6c89a6-83ae-4e3a-b10b-6da898e4857b","Type":"ContainerStarted","Data":"ba6ef5f60b6bb55ed6ac8f7461c676a3d09b5d28692464a81de20dd1c2629baa"}
Nov 03 23:00:19.806427 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:19.806409    4431 generic.go:356] "Generic (PLEG): container finished" podID="becaf054-eec7-43bb-8443-8d5052e5859a" containerID="dc134a7b64f06237e11feec6bfd17b48f0775c0e79bd55f26c9cc9a3a1817c39" exitCode=60
Nov 03 23:00:19.806516 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:19.806471    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5187e1-29895780-j9x65" event={"ID":"becaf054-eec7-43bb-8443-8d5052e5859a","Type":"ContainerDied","Data":"dc134a7b64f06237e11feec6bfd17b48f0775c0e79bd55f26c9cc9a3a1817c39"}
Nov 03 23:00:19.806770 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:19.806741    4431 scope.go:122] "RemoveContainer" containerID="dc134a7b64f06237e11feec6bfd17b48f0775c0e79bd55f26c9cc9a3a1817c39"
Nov 03 23:00:19.806999 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:00:19.806970    4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-5187e1-29895780-j9x65_odh-ai-gateway-infra(becaf054-eec7-43bb-8443-8d5052e5859a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5187e1-29895780-j9x65" podUID="becaf054-eec7-43bb-8443-8d5052e5859a"
Nov 03 23:00:19.811058 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:19.811007    4431 scope.go:122] "RemoveContainer" containerID="93fbc9fe96465e40d689abceb7e666d243576006b5f6a6473455dbc661f6731f"
Nov 03 23:00:19.817367 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:19.817344    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:00:19.873239 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:19.872468    4431 scope.go:122] "RemoveContainer" containerID="32f6c1b51d83c90359251121459ce7bbd50836b4a8d609b836bdff62f0930320"
Nov 03 23:00:19.883089 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:19.883054    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5187e1-29895780-j9x65"]
Nov 03 23:00:20.015549 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:20.015519    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e19850-29895780-z748r"]
Nov 03 23:00:20.064332 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:20.062434    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895780-dbtlc"
Nov 03 23:00:20.068918 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:20.068886    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-23d281-29895780-cbhhq"]
Nov 03 23:00:20.090998 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:20.090967    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bfb8b9d-1ad7-4c94-96ab-25f74946da5b" path="/var/lib/kubelet/pods/8bfb8b9d-1ad7-4c94-96ab-25f74946da5b/volumes"
Nov 03 23:00:20.136546 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:20.136511    4431 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Nov 03 23:00:20.136718 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:20.136569    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm" podUID="44002d96-03c6-4063-b992-a531d4fe8016" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Nov 03 23:00:20.190630 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:20.190597    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/291a5765-87a6-46b2-88f4-1d4cd9056fab-kube-api-access-lxl56\" (UniqueName: \"kubernetes.io/projected/291a5765-87a6-46b2-88f4-1d4cd9056fab-kube-api-access-lxl56\") pod \"291a5765-87a6-46b2-88f4-1d4cd9056fab\" (UID: \"291a5765-87a6-46b2-88f4-1d4cd9056fab\") "
Nov 03 23:00:20.190799 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:20.190718    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/291a5765-87a6-46b2-88f4-1d4cd9056fab-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/291a5765-87a6-46b2-88f4-1d4cd9056fab-maas-api-service-ca\") pod \"291a5765-87a6-46b2-88f4-1d4cd9056fab\" (UID: \"291a5765-87a6-46b2-88f4-1d4cd9056fab\") "
Nov 03 23:00:20.191128 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:20.191101    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/291a5765-87a6-46b2-88f4-1d4cd9056fab-maas-api-service-ca" pod "291a5765-87a6-46b2-88f4-1d4cd9056fab" (UID: "291a5765-87a6-46b2-88f4-1d4cd9056fab"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:00:20.191409 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:20.191391    4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/291a5765-87a6-46b2-88f4-1d4cd9056fab-maas-api-service-ca\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:00:20.192835 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:20.192810    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/291a5765-87a6-46b2-88f4-1d4cd9056fab-kube-api-access-lxl56" pod "291a5765-87a6-46b2-88f4-1d4cd9056fab" (UID: "291a5765-87a6-46b2-88f4-1d4cd9056fab"). InnerVolumeSpecName "kube-api-access-lxl56". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:00:20.292509 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:20.292473    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxl56\" (UniqueName: \"kubernetes.io/projected/291a5765-87a6-46b2-88f4-1d4cd9056fab-kube-api-access-lxl56\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:00:20.820071 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:20.820049    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895780-dbtlc"
Nov 03 23:00:20.820071 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:20.820057    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895780-dbtlc" event={"ID":"291a5765-87a6-46b2-88f4-1d4cd9056fab","Type":"ContainerDied","Data":"ee9e88b01d35a99a56696c1aa8d6b5927f14efb3d9daf9b201aae4a800d0fbff"}
Nov 03 23:00:20.820524 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:20.820142    4431 scope.go:122] "RemoveContainer" containerID="5ccbadaf1a38a60332fd44fb37c0dad71ece2bd253965fa627e120686d03d93f"
Nov 03 23:00:20.822513 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:20.822486    4431 generic.go:356] "Generic (PLEG): container finished" podID="04763bf4-4b53-41a0-a47d-72d46f1f9563" containerID="b4f789269b1f142b238e9b9ac4e1a37c3966b1f0f4adb332d7533db473b45f5b" exitCode=60
Nov 03 23:00:20.822690 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:20.822521    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-ff80c0-29895780-srjhc" event={"ID":"04763bf4-4b53-41a0-a47d-72d46f1f9563","Type":"ContainerDied","Data":"b4f789269b1f142b238e9b9ac4e1a37c3966b1f0f4adb332d7533db473b45f5b"}
Nov 03 23:00:20.822994 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:20.822974    4431 scope.go:122] "RemoveContainer" containerID="b4f789269b1f142b238e9b9ac4e1a37c3966b1f0f4adb332d7533db473b45f5b"
Nov 03 23:00:20.823376 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:00:20.823338    4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-ff80c0-29895780-srjhc_odh-ai-gateway-infra(04763bf4-4b53-41a0-a47d-72d46f1f9563)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-ff80c0-29895780-srjhc" podUID="04763bf4-4b53-41a0-a47d-72d46f1f9563"
Nov 03 23:00:20.832664 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:20.832642    4431 scope.go:122] "RemoveContainer" containerID="5a8cb603a1e15215cd6b26d9d7dd5c15ffc09a38191a588409ce7fc3b55849b6"
Nov 03 23:00:20.833886 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:20.833776    4431 generic.go:356] "Generic (PLEG): container finished" podID="1f01190d-0246-487f-a9af-54dab722b3e3" containerID="601876aadd7f8ddaa8440c65a46fef521560a6648bc576ddea9026690f2ababe" exitCode=60
Nov 03 23:00:20.833886 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:20.833791    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3920cb-29895780-bhr6s" event={"ID":"1f01190d-0246-487f-a9af-54dab722b3e3","Type":"ContainerDied","Data":"601876aadd7f8ddaa8440c65a46fef521560a6648bc576ddea9026690f2ababe"}
Nov 03 23:00:20.834282 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:20.834264    4431 scope.go:122] "RemoveContainer" containerID="601876aadd7f8ddaa8440c65a46fef521560a6648bc576ddea9026690f2ababe"
Nov 03 23:00:20.834582 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:00:20.834559    4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-3920cb-29895780-bhr6s_odh-ai-gateway-infra(1f01190d-0246-487f-a9af-54dab722b3e3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3920cb-29895780-bhr6s" podUID="1f01190d-0246-487f-a9af-54dab722b3e3"
Nov 03 23:00:20.835895 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:20.835875    4431 generic.go:356] "Generic (PLEG): container finished" podID="9f6c89a6-83ae-4e3a-b10b-6da898e4857b" containerID="ba6ef5f60b6bb55ed6ac8f7461c676a3d09b5d28692464a81de20dd1c2629baa" exitCode=60
Nov 03 23:00:20.835999 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:20.835947    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895780-dbslb" event={"ID":"9f6c89a6-83ae-4e3a-b10b-6da898e4857b","Type":"ContainerDied","Data":"ba6ef5f60b6bb55ed6ac8f7461c676a3d09b5d28692464a81de20dd1c2629baa"}
Nov 03 23:00:20.836430 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:20.836412    4431 scope.go:122] "RemoveContainer" containerID="ba6ef5f60b6bb55ed6ac8f7461c676a3d09b5d28692464a81de20dd1c2629baa"
Nov 03 23:00:20.836692 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:00:20.836674    4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895780-dbslb_odh-ai-gateway-infra(9f6c89a6-83ae-4e3a-b10b-6da898e4857b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895780-dbslb" podUID="9f6c89a6-83ae-4e3a-b10b-6da898e4857b"
Nov 03 23:00:20.871372 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:20.871147    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-ff80c0-29895780-srjhc"]
Nov 03 23:00:20.883530 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:20.883499    4431 scope.go:122] "RemoveContainer" containerID="0f257f005e164d175b92d4e1fe3a9bc7ace2b13830871de954474a9781f6fe48"
Nov 03 23:00:20.904496 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:20.904463    4431 scope.go:122] "RemoveContainer" containerID="0d35fa8f74cbcbd50b40714b98050b657f9cd5fc8eb2e01f20caf6c422f9c2f4"
Nov 03 23:00:21.011496 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:21.011464    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895780-dbtlc"]
Nov 03 23:00:21.020964 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:21.020564    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895780-dbtlc"]
Nov 03 23:00:21.036229 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:21.036182    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3920cb-29895780-bhr6s"]
Nov 03 23:00:21.071942 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:21.066478    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-746ccb70-wxzpp"]
Nov 03 23:00:21.075247 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:21.072611    4431 cpu_manager.go:414] "containerMap: removing container" podUID="291a5765-87a6-46b2-88f4-1d4cd9056fab" containerName="cleanup"
Nov 03 23:00:21.075247 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:21.072639    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="291a5765-87a6-46b2-88f4-1d4cd9056fab" containerName="cleanup"
Nov 03 23:00:21.075247 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:21.072656    4431 cpu_manager.go:414] "containerMap: removing container" podUID="8bfb8b9d-1ad7-4c94-96ab-25f74946da5b" containerName="cleanup"
Nov 03 23:00:21.075247 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:21.072665    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bfb8b9d-1ad7-4c94-96ab-25f74946da5b" containerName="cleanup"
Nov 03 23:00:21.075247 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:21.072678    4431 cpu_manager.go:414] "containerMap: removing container" podUID="291a5765-87a6-46b2-88f4-1d4cd9056fab" containerName="cleanup"
Nov 03 23:00:21.075247 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:21.072685    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="291a5765-87a6-46b2-88f4-1d4cd9056fab" containerName="cleanup"
Nov 03 23:00:21.075247 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:21.072704    4431 cpu_manager.go:414] "containerMap: removing container" podUID="291a5765-87a6-46b2-88f4-1d4cd9056fab" containerName="cleanup"
Nov 03 23:00:21.075247 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:21.072711    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="291a5765-87a6-46b2-88f4-1d4cd9056fab" containerName="cleanup"
Nov 03 23:00:21.075247 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:21.072772    4431 cpu_manager.go:414] "containerMap: removing container" podUID="8bfb8b9d-1ad7-4c94-96ab-25f74946da5b" containerName="cleanup"
Nov 03 23:00:21.075247 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:21.072779    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bfb8b9d-1ad7-4c94-96ab-25f74946da5b" containerName="cleanup"
Nov 03 23:00:21.075247 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:21.072801    4431 cpu_manager.go:414] "containerMap: removing container" podUID="8bfb8b9d-1ad7-4c94-96ab-25f74946da5b" containerName="cleanup"
Nov 03 23:00:21.075247 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:21.072807    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bfb8b9d-1ad7-4c94-96ab-25f74946da5b" containerName="cleanup"
Nov 03 23:00:21.075247 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:21.073003    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="291a5765-87a6-46b2-88f4-1d4cd9056fab" containerName="cleanup"
Nov 03 23:00:21.075247 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:21.073015    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bfb8b9d-1ad7-4c94-96ab-25f74946da5b" containerName="cleanup"
Nov 03 23:00:21.075247 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:21.073025    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="291a5765-87a6-46b2-88f4-1d4cd9056fab" containerName="cleanup"
Nov 03 23:00:21.075247 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:21.073044    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bfb8b9d-1ad7-4c94-96ab-25f74946da5b" containerName="cleanup"
Nov 03 23:00:21.075247 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:21.073056    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bfb8b9d-1ad7-4c94-96ab-25f74946da5b" containerName="cleanup"
Nov 03 23:00:21.111890 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:21.111866    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895780-dbslb"]
Nov 03 23:00:21.112087 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:21.112075    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-746ccb70-wxzpp"]
Nov 03 23:00:21.112278 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:21.112265    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-746ccb70-wxzpp"
Nov 03 23:00:21.112647 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:21.112623    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5187e1-29895780-j9x65"
Nov 03 23:00:21.129524 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:21.129496    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 23:00:21.136524 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:21.136494    4431 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Nov 03 23:00:21.136668 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:21.136550    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm" podUID="44002d96-03c6-4063-b992-a531d4fe8016" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Nov 03 23:00:21.199369 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:21.199346    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e19850-29895780-z748r"
Nov 03 23:00:21.207722 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:21.207698    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/becaf054-eec7-43bb-8443-8d5052e5859a-kube-api-access-qdjcc\" (UniqueName: \"kubernetes.io/projected/becaf054-eec7-43bb-8443-8d5052e5859a-kube-api-access-qdjcc\") pod \"becaf054-eec7-43bb-8443-8d5052e5859a\" (UID: \"becaf054-eec7-43bb-8443-8d5052e5859a\") "
Nov 03 23:00:21.207838 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:21.207761    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/becaf054-eec7-43bb-8443-8d5052e5859a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/becaf054-eec7-43bb-8443-8d5052e5859a-maas-api-service-ca\") pod \"becaf054-eec7-43bb-8443-8d5052e5859a\" (UID: \"becaf054-eec7-43bb-8443-8d5052e5859a\") "
Nov 03 23:00:21.208062 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:21.207982    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-23d281-29895780-cbhhq"
Nov 03 23:00:21.208181 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:21.208035    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5tk8\" (UniqueName: \"kubernetes.io/projected/5ac3bb0e-6b8e-4b71-be4d-06fb3a2135b8-kube-api-access-r5tk8\") pod \"maas-api-revoke-sub-746ccb70-wxzpp\" (UID: \"5ac3bb0e-6b8e-4b71-be4d-06fb3a2135b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-746ccb70-wxzpp"
Nov 03 23:00:21.208181 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:21.208169    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/becaf054-eec7-43bb-8443-8d5052e5859a-maas-api-service-ca" pod "becaf054-eec7-43bb-8443-8d5052e5859a" (UID: "becaf054-eec7-43bb-8443-8d5052e5859a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:00:21.208317 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:21.208273    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ac3bb0e-6b8e-4b71-be4d-06fb3a2135b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-746ccb70-wxzpp\" (UID: \"5ac3bb0e-6b8e-4b71-be4d-06fb3a2135b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-746ccb70-wxzpp"
Nov 03 23:00:21.208448 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:21.208427    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ac3bb0e-6b8e-4b71-be4d-06fb3a2135b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-746ccb70-wxzpp\" (UID: \"5ac3bb0e-6b8e-4b71-be4d-06fb3a2135b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-746ccb70-wxzpp"
Nov 03 23:00:21.208639 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:21.208622    4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/becaf054-eec7-43bb-8443-8d5052e5859a-maas-api-service-ca\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:00:21.209874 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:21.209858    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/becaf054-eec7-43bb-8443-8d5052e5859a-kube-api-access-qdjcc" pod "becaf054-eec7-43bb-8443-8d5052e5859a" (UID: "becaf054-eec7-43bb-8443-8d5052e5859a"). InnerVolumeSpecName "kube-api-access-qdjcc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:00:21.309160 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:21.309119    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c362bb2-ed25-42d4-8907-7ea5985b3df4-kube-api-access-d57hd\" (UniqueName: \"kubernetes.io/projected/1c362bb2-ed25-42d4-8907-7ea5985b3df4-kube-api-access-d57hd\") pod \"1c362bb2-ed25-42d4-8907-7ea5985b3df4\" (UID: \"1c362bb2-ed25-42d4-8907-7ea5985b3df4\") "
Nov 03 23:00:21.309354 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:21.309306    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6d9effb-0fd1-4443-b1ee-82600bd004ff-kube-api-access-rrqbx\" (UniqueName: \"kubernetes.io/projected/b6d9effb-0fd1-4443-b1ee-82600bd004ff-kube-api-access-rrqbx\") pod \"b6d9effb-0fd1-4443-b1ee-82600bd004ff\" (UID: \"b6d9effb-0fd1-4443-b1ee-82600bd004ff\") "
Nov 03 23:00:21.309446 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:21.309369    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c362bb2-ed25-42d4-8907-7ea5985b3df4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c362bb2-ed25-42d4-8907-7ea5985b3df4-maas-api-service-ca\") pod \"1c362bb2-ed25-42d4-8907-7ea5985b3df4\" (UID: \"1c362bb2-ed25-42d4-8907-7ea5985b3df4\") "
Nov 03 23:00:21.309900 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:21.309867    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c362bb2-ed25-42d4-8907-7ea5985b3df4-maas-api-service-ca" pod "1c362bb2-ed25-42d4-8907-7ea5985b3df4" (UID: "1c362bb2-ed25-42d4-8907-7ea5985b3df4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:00:21.310311 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:21.310267    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6d9effb-0fd1-4443-b1ee-82600bd004ff-maas-api-service-ca" pod "b6d9effb-0fd1-4443-b1ee-82600bd004ff" (UID: "b6d9effb-0fd1-4443-b1ee-82600bd004ff"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:00:21.310618 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:21.310580    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6d9effb-0fd1-4443-b1ee-82600bd004ff-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6d9effb-0fd1-4443-b1ee-82600bd004ff-maas-api-service-ca\") pod \"b6d9effb-0fd1-4443-b1ee-82600bd004ff\" (UID: \"b6d9effb-0fd1-4443-b1ee-82600bd004ff\") "
Nov 03 23:00:21.311368 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:21.311077    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5tk8\" (UniqueName: \"kubernetes.io/projected/5ac3bb0e-6b8e-4b71-be4d-06fb3a2135b8-kube-api-access-r5tk8\") pod \"maas-api-revoke-sub-746ccb70-wxzpp\" (UID: \"5ac3bb0e-6b8e-4b71-be4d-06fb3a2135b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-746ccb70-wxzpp"
Nov 03 23:00:21.311368 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:21.311151    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ac3bb0e-6b8e-4b71-be4d-06fb3a2135b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-746ccb70-wxzpp\" (UID: \"5ac3bb0e-6b8e-4b71-be4d-06fb3a2135b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-746ccb70-wxzpp"
Nov 03 23:00:21.311368 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:21.311277    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ac3bb0e-6b8e-4b71-be4d-06fb3a2135b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-746ccb70-wxzpp\" (UID: \"5ac3bb0e-6b8e-4b71-be4d-06fb3a2135b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-746ccb70-wxzpp"
Nov 03 23:00:21.311554 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:21.311377    4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6d9effb-0fd1-4443-b1ee-82600bd004ff-maas-api-service-ca\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:00:21.311554 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:21.311390    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdjcc\" (UniqueName: \"kubernetes.io/projected/becaf054-eec7-43bb-8443-8d5052e5859a-kube-api-access-qdjcc\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:00:21.311554 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:21.311407    4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c362bb2-ed25-42d4-8907-7ea5985b3df4-maas-api-service-ca\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:00:21.312110 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:21.311837    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c362bb2-ed25-42d4-8907-7ea5985b3df4-kube-api-access-d57hd" pod "1c362bb2-ed25-42d4-8907-7ea5985b3df4" (UID: "1c362bb2-ed25-42d4-8907-7ea5985b3df4"). InnerVolumeSpecName "kube-api-access-d57hd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:00:21.312253 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:21.312116    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6d9effb-0fd1-4443-b1ee-82600bd004ff-kube-api-access-rrqbx" pod "b6d9effb-0fd1-4443-b1ee-82600bd004ff" (UID: "b6d9effb-0fd1-4443-b1ee-82600bd004ff"). InnerVolumeSpecName "kube-api-access-rrqbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:00:21.312880 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:21.312473    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ac3bb0e-6b8e-4b71-be4d-06fb3a2135b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-746ccb70-wxzpp\" (UID: \"5ac3bb0e-6b8e-4b71-be4d-06fb3a2135b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-746ccb70-wxzpp"
Nov 03 23:00:21.314352 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:21.314329    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ac3bb0e-6b8e-4b71-be4d-06fb3a2135b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-746ccb70-wxzpp\" (UID: \"5ac3bb0e-6b8e-4b71-be4d-06fb3a2135b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-746ccb70-wxzpp"
Nov 03 23:00:21.334806 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:21.334741    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5tk8\" (UniqueName: \"kubernetes.io/projected/5ac3bb0e-6b8e-4b71-be4d-06fb3a2135b8-kube-api-access-r5tk8\") pod \"maas-api-revoke-sub-746ccb70-wxzpp\" (UID: \"5ac3bb0e-6b8e-4b71-be4d-06fb3a2135b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-746ccb70-wxzpp"
Nov 03 23:00:21.413252 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:21.413195    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d57hd\" (UniqueName: \"kubernetes.io/projected/1c362bb2-ed25-42d4-8907-7ea5985b3df4-kube-api-access-d57hd\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:00:21.413252 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:21.413243    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrqbx\" (UniqueName: \"kubernetes.io/projected/b6d9effb-0fd1-4443-b1ee-82600bd004ff-kube-api-access-rrqbx\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:00:21.505907 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:21.505869    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-746ccb70-wxzpp"
Nov 03 23:00:21.524139 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:21.524110    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:00:21.871618 ip-10-0-139-73 kubenswrapper[4431]: W1103 23:00:21.871585    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ac3bb0e_6b8e_4b71_be4d_06fb3a2135b8.slice/crio-63ee348043137cb842461279324a180d64adf7af6efa2386c675e80b1abe94ae WatchSource:0}: Error finding container 63ee348043137cb842461279324a180d64adf7af6efa2386c675e80b1abe94ae: Status 404 returned error can't find the container with id 63ee348043137cb842461279324a180d64adf7af6efa2386c675e80b1abe94ae
Nov 03 23:00:21.873333 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:21.873305    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-746ccb70-wxzpp"]
Nov 03 23:00:21.876642 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:21.876621    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5187e1-29895780-j9x65"
Nov 03 23:00:21.876751 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:21.876624    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5187e1-29895780-j9x65" event={"ID":"becaf054-eec7-43bb-8443-8d5052e5859a","Type":"ContainerDied","Data":"d52ac3f219092260c54c600a2060a315a9de9cb055012489ffb1675a1cc41267"}
Nov 03 23:00:21.877039 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:21.877027    4431 scope.go:122] "RemoveContainer" containerID="dc134a7b64f06237e11feec6bfd17b48f0775c0e79bd55f26c9cc9a3a1817c39"
Nov 03 23:00:21.884658 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:21.884634    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e19850-29895780-z748r" event={"ID":"b6d9effb-0fd1-4443-b1ee-82600bd004ff","Type":"ContainerDied","Data":"ec1be7b3f176823e9eef145b3c219afad1f596bf27c7879647186b4e92c749a8"}
Nov 03 23:00:21.884870 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:21.884854    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e19850-29895780-z748r"
Nov 03 23:00:21.886585 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:21.886561    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-23d281-29895780-cbhhq" event={"ID":"1c362bb2-ed25-42d4-8907-7ea5985b3df4","Type":"ContainerDied","Data":"1aa03c830ee223d5636c75c90ae8fd45fa5d78605085abb47077546ebd568962"}
Nov 03 23:00:21.886759 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:21.886741    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-23d281-29895780-cbhhq"
Nov 03 23:00:21.913001 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:21.912971    4431 scope.go:122] "RemoveContainer" containerID="e23e406f8f5b63623666f81d3bdc5f1dc561e2f225c85a8ef21340b801dbdcb9"
Nov 03 23:00:21.927735 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:21.927694    4431 scope.go:122] "RemoveContainer" containerID="91cdc4fcab5fca1ef64e04abe1c37f7e6e23e22c95bf2a27e2d9dbe28893ea78"
Nov 03 23:00:21.981881 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:21.980461    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-d8a3d1/auth-test-model-d8a3d1-kserve-5fbc7778b9-kj9kw" podUID="1503c14c-4dab-435d-b29f-5c60d0c0acfb" containerName="main" containerID="cri-o://b28828735935672e686447cafd01882daefadfbcdba667ebeff32f5b0f4f94fd" gracePeriod=45
Nov 03 23:00:22.091070 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:22.090898    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="291a5765-87a6-46b2-88f4-1d4cd9056fab" path="/var/lib/kubelet/pods/291a5765-87a6-46b2-88f4-1d4cd9056fab/volumes"
Nov 03 23:00:22.177596 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:22.177537    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e19850-29895780-z748r"]
Nov 03 23:00:22.184944 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:22.178751    4431 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Nov 03 23:00:22.184944 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:22.178808    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm" podUID="44002d96-03c6-4063-b992-a531d4fe8016" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Nov 03 23:00:22.184944 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:22.182174    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e19850-29895780-z748r"]
Nov 03 23:00:22.203724 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:22.201239    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-23d281-29895780-cbhhq"]
Nov 03 23:00:22.215337 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:22.215268    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-23d281-29895780-cbhhq"]
Nov 03 23:00:22.277129 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:22.277093    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5187e1-29895780-j9x65"]
Nov 03 23:00:22.284678 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:22.284649    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5187e1-29895780-j9x65"]
Nov 03 23:00:22.290182 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:22.290160    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-ff80c0-29895780-srjhc"
Nov 03 23:00:22.293710 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:22.293686    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3920cb-29895780-bhr6s"
Nov 03 23:00:22.297321 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:22.296964    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895780-dbslb"
Nov 03 23:00:22.481076 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:22.481036    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f01190d-0246-487f-a9af-54dab722b3e3-kube-api-access-zsd5f\" (UniqueName: \"kubernetes.io/projected/1f01190d-0246-487f-a9af-54dab722b3e3-kube-api-access-zsd5f\") pod \"1f01190d-0246-487f-a9af-54dab722b3e3\" (UID: \"1f01190d-0246-487f-a9af-54dab722b3e3\") "
Nov 03 23:00:22.481269 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:22.481247    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f6c89a6-83ae-4e3a-b10b-6da898e4857b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f6c89a6-83ae-4e3a-b10b-6da898e4857b-maas-api-service-ca\") pod \"9f6c89a6-83ae-4e3a-b10b-6da898e4857b\" (UID: \"9f6c89a6-83ae-4e3a-b10b-6da898e4857b\") "
Nov 03 23:00:22.481363 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:22.481305    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04763bf4-4b53-41a0-a47d-72d46f1f9563-kube-api-access-pcpgp\" (UniqueName: \"kubernetes.io/projected/04763bf4-4b53-41a0-a47d-72d46f1f9563-kube-api-access-pcpgp\") pod \"04763bf4-4b53-41a0-a47d-72d46f1f9563\" (UID: \"04763bf4-4b53-41a0-a47d-72d46f1f9563\") "
Nov 03 23:00:22.481363 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:22.481344    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04763bf4-4b53-41a0-a47d-72d46f1f9563-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04763bf4-4b53-41a0-a47d-72d46f1f9563-maas-api-service-ca\") pod \"04763bf4-4b53-41a0-a47d-72d46f1f9563\" (UID: \"04763bf4-4b53-41a0-a47d-72d46f1f9563\") "
Nov 03 23:00:22.481460 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:22.481423    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f6c89a6-83ae-4e3a-b10b-6da898e4857b-kube-api-access-6cm9g\" (UniqueName: \"kubernetes.io/projected/9f6c89a6-83ae-4e3a-b10b-6da898e4857b-kube-api-access-6cm9g\") pod \"9f6c89a6-83ae-4e3a-b10b-6da898e4857b\" (UID: \"9f6c89a6-83ae-4e3a-b10b-6da898e4857b\") "
Nov 03 23:00:22.481516 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:22.481494    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f01190d-0246-487f-a9af-54dab722b3e3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f01190d-0246-487f-a9af-54dab722b3e3-maas-api-service-ca\") pod \"1f01190d-0246-487f-a9af-54dab722b3e3\" (UID: \"1f01190d-0246-487f-a9af-54dab722b3e3\") "
Nov 03 23:00:22.482671 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:22.482642    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d8a3d1/auth-test-model-d8a3d1-kserve-5fbc7778b9-kj9kw"
Nov 03 23:00:22.482803 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:22.482717    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f6c89a6-83ae-4e3a-b10b-6da898e4857b-maas-api-service-ca" pod "9f6c89a6-83ae-4e3a-b10b-6da898e4857b" (UID: "9f6c89a6-83ae-4e3a-b10b-6da898e4857b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:00:22.483023 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:22.482986    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04763bf4-4b53-41a0-a47d-72d46f1f9563-maas-api-service-ca" pod "04763bf4-4b53-41a0-a47d-72d46f1f9563" (UID: "04763bf4-4b53-41a0-a47d-72d46f1f9563"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:00:22.484267 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:22.484239    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f01190d-0246-487f-a9af-54dab722b3e3-maas-api-service-ca" pod "1f01190d-0246-487f-a9af-54dab722b3e3" (UID: "1f01190d-0246-487f-a9af-54dab722b3e3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:00:22.487576 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:22.487517    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f01190d-0246-487f-a9af-54dab722b3e3-kube-api-access-zsd5f" pod "1f01190d-0246-487f-a9af-54dab722b3e3" (UID: "1f01190d-0246-487f-a9af-54dab722b3e3"). InnerVolumeSpecName "kube-api-access-zsd5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:00:22.487751 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:22.487731    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04763bf4-4b53-41a0-a47d-72d46f1f9563-kube-api-access-pcpgp" pod "04763bf4-4b53-41a0-a47d-72d46f1f9563" (UID: "04763bf4-4b53-41a0-a47d-72d46f1f9563"). InnerVolumeSpecName "kube-api-access-pcpgp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:00:22.489295 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:22.489265    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f6c89a6-83ae-4e3a-b10b-6da898e4857b-kube-api-access-6cm9g" pod "9f6c89a6-83ae-4e3a-b10b-6da898e4857b" (UID: "9f6c89a6-83ae-4e3a-b10b-6da898e4857b"). InnerVolumeSpecName "kube-api-access-6cm9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:00:22.583384 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:22.583348    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1503c14c-4dab-435d-b29f-5c60d0c0acfb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1503c14c-4dab-435d-b29f-5c60d0c0acfb-dshm\") pod \"1503c14c-4dab-435d-b29f-5c60d0c0acfb\" (UID: \"1503c14c-4dab-435d-b29f-5c60d0c0acfb\") "
Nov 03 23:00:22.583596 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:22.583424    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1503c14c-4dab-435d-b29f-5c60d0c0acfb-home\" (UniqueName: \"kubernetes.io/empty-dir/1503c14c-4dab-435d-b29f-5c60d0c0acfb-home\") pod \"1503c14c-4dab-435d-b29f-5c60d0c0acfb\" (UID: \"1503c14c-4dab-435d-b29f-5c60d0c0acfb\") "
Nov 03 23:00:22.583596 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:22.583534    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1503c14c-4dab-435d-b29f-5c60d0c0acfb-tls-certs\" (UniqueName: \"kubernetes.io/secret/1503c14c-4dab-435d-b29f-5c60d0c0acfb-tls-certs\") pod \"1503c14c-4dab-435d-b29f-5c60d0c0acfb\" (UID: \"1503c14c-4dab-435d-b29f-5c60d0c0acfb\") "
Nov 03 23:00:22.583596 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:22.583581    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1503c14c-4dab-435d-b29f-5c60d0c0acfb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1503c14c-4dab-435d-b29f-5c60d0c0acfb-tmp-dir\") pod \"1503c14c-4dab-435d-b29f-5c60d0c0acfb\" (UID: \"1503c14c-4dab-435d-b29f-5c60d0c0acfb\") "
Nov 03 23:00:22.583771 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:22.583633    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1503c14c-4dab-435d-b29f-5c60d0c0acfb-kube-api-access-95g99\" (UniqueName: \"kubernetes.io/projected/1503c14c-4dab-435d-b29f-5c60d0c0acfb-kube-api-access-95g99\") pod \"1503c14c-4dab-435d-b29f-5c60d0c0acfb\" (UID: \"1503c14c-4dab-435d-b29f-5c60d0c0acfb\") "
Nov 03 23:00:22.583824 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:22.583768    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1503c14c-4dab-435d-b29f-5c60d0c0acfb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1503c14c-4dab-435d-b29f-5c60d0c0acfb-model-cache\") pod \"1503c14c-4dab-435d-b29f-5c60d0c0acfb\" (UID: \"1503c14c-4dab-435d-b29f-5c60d0c0acfb\") "
Nov 03 23:00:22.584343 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:22.584295    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1503c14c-4dab-435d-b29f-5c60d0c0acfb-tmp-dir" pod "1503c14c-4dab-435d-b29f-5c60d0c0acfb" (UID: "1503c14c-4dab-435d-b29f-5c60d0c0acfb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:00:22.584690 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:22.584667    4431 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1503c14c-4dab-435d-b29f-5c60d0c0acfb-tmp-dir\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:00:22.584819 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:22.584795    4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f01190d-0246-487f-a9af-54dab722b3e3-maas-api-service-ca\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:00:22.584819 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:22.584820    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsd5f\" (UniqueName: \"kubernetes.io/projected/1f01190d-0246-487f-a9af-54dab722b3e3-kube-api-access-zsd5f\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:00:22.585096 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:22.584833    4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f6c89a6-83ae-4e3a-b10b-6da898e4857b-maas-api-service-ca\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:00:22.585096 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:22.584846    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcpgp\" (UniqueName: \"kubernetes.io/projected/04763bf4-4b53-41a0-a47d-72d46f1f9563-kube-api-access-pcpgp\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:00:22.585096 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:22.584858    4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04763bf4-4b53-41a0-a47d-72d46f1f9563-maas-api-service-ca\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:00:22.585096 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:22.584870    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cm9g\" (UniqueName: \"kubernetes.io/projected/9f6c89a6-83ae-4e3a-b10b-6da898e4857b-kube-api-access-6cm9g\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:00:22.585096 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:22.584694    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1503c14c-4dab-435d-b29f-5c60d0c0acfb-model-cache" pod "1503c14c-4dab-435d-b29f-5c60d0c0acfb" (UID: "1503c14c-4dab-435d-b29f-5c60d0c0acfb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:00:22.587492 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:22.585117    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1503c14c-4dab-435d-b29f-5c60d0c0acfb-home" pod "1503c14c-4dab-435d-b29f-5c60d0c0acfb" (UID: "1503c14c-4dab-435d-b29f-5c60d0c0acfb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:00:22.588096 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:22.588063    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1503c14c-4dab-435d-b29f-5c60d0c0acfb-kube-api-access-95g99" pod "1503c14c-4dab-435d-b29f-5c60d0c0acfb" (UID: "1503c14c-4dab-435d-b29f-5c60d0c0acfb"). InnerVolumeSpecName "kube-api-access-95g99". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:00:22.588096 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:22.588081    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1503c14c-4dab-435d-b29f-5c60d0c0acfb-tls-certs" pod "1503c14c-4dab-435d-b29f-5c60d0c0acfb" (UID: "1503c14c-4dab-435d-b29f-5c60d0c0acfb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:00:22.588435 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:22.588411    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1503c14c-4dab-435d-b29f-5c60d0c0acfb-dshm" pod "1503c14c-4dab-435d-b29f-5c60d0c0acfb" (UID: "1503c14c-4dab-435d-b29f-5c60d0c0acfb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:00:22.686682 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:22.686577    4431 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1503c14c-4dab-435d-b29f-5c60d0c0acfb-tls-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:00:22.686682 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:22.686620    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95g99\" (UniqueName: \"kubernetes.io/projected/1503c14c-4dab-435d-b29f-5c60d0c0acfb-kube-api-access-95g99\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:00:22.686682 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:22.686635    4431 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1503c14c-4dab-435d-b29f-5c60d0c0acfb-model-cache\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:00:22.686682 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:22.686648    4431 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1503c14c-4dab-435d-b29f-5c60d0c0acfb-dshm\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:00:22.686682 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:22.686661    4431 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1503c14c-4dab-435d-b29f-5c60d0c0acfb-home\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:00:22.919613 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:22.919575    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-ff80c0-29895780-srjhc" event={"ID":"04763bf4-4b53-41a0-a47d-72d46f1f9563","Type":"ContainerDied","Data":"d828cadd76775f3db5d4ea1137f0c6860e9c976618514dec6ae3ec1507f873fc"}
Nov 03 23:00:22.920113 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:22.919587    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-ff80c0-29895780-srjhc"
Nov 03 23:00:22.920113 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:22.919674    4431 scope.go:122] "RemoveContainer" containerID="b4f789269b1f142b238e9b9ac4e1a37c3966b1f0f4adb332d7533db473b45f5b"
Nov 03 23:00:22.923825 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:22.923796    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-746ccb70-wxzpp" event={"ID":"5ac3bb0e-6b8e-4b71-be4d-06fb3a2135b8","Type":"ContainerStarted","Data":"5743d436771d83fb1a53146f7f3ccd9f91edd6d7e37618f8241ba0ab25411c46"}
Nov 03 23:00:22.923983 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:22.923963    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-746ccb70-wxzpp" event={"ID":"5ac3bb0e-6b8e-4b71-be4d-06fb3a2135b8","Type":"ContainerStarted","Data":"63ee348043137cb842461279324a180d64adf7af6efa2386c675e80b1abe94ae"}
Nov 03 23:00:22.925270 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:22.925244    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3920cb-29895780-bhr6s" event={"ID":"1f01190d-0246-487f-a9af-54dab722b3e3","Type":"ContainerDied","Data":"a68f9a69318fab07b6cc5a736315f2743bc476de0649f28752f5378fc3c9d467"}
Nov 03 23:00:22.925389 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:22.925276    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3920cb-29895780-bhr6s"
Nov 03 23:00:22.926712 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:22.926693    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895780-dbslb"
Nov 03 23:00:22.926818 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:22.926735    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895780-dbslb" event={"ID":"9f6c89a6-83ae-4e3a-b10b-6da898e4857b","Type":"ContainerDied","Data":"8398000f3c3c71f909b6fd8f29938fbdd69e29bba3cae3fa284307c2e431a88f"}
Nov 03 23:00:22.928270 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:22.928241    4431 generic.go:356] "Generic (PLEG): container finished" podID="1503c14c-4dab-435d-b29f-5c60d0c0acfb" containerID="b28828735935672e686447cafd01882daefadfbcdba667ebeff32f5b0f4f94fd" exitCode=0
Nov 03 23:00:22.928366 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:22.928333    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d8a3d1/auth-test-model-d8a3d1-kserve-5fbc7778b9-kj9kw" event={"ID":"1503c14c-4dab-435d-b29f-5c60d0c0acfb","Type":"ContainerDied","Data":"b28828735935672e686447cafd01882daefadfbcdba667ebeff32f5b0f4f94fd"}
Nov 03 23:00:22.928366 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:22.928339    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d8a3d1/auth-test-model-d8a3d1-kserve-5fbc7778b9-kj9kw"
Nov 03 23:00:22.928366 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:22.928360    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d8a3d1/auth-test-model-d8a3d1-kserve-5fbc7778b9-kj9kw" event={"ID":"1503c14c-4dab-435d-b29f-5c60d0c0acfb","Type":"ContainerDied","Data":"4b407ea464c5f36b7c82c53ab6f7b342f029c959769a2fb6dfda985caeae3d9c"}
Nov 03 23:00:22.937451 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:22.937362    4431 scope.go:122] "RemoveContainer" containerID="601876aadd7f8ddaa8440c65a46fef521560a6648bc576ddea9026690f2ababe"
Nov 03 23:00:22.955782 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:22.955757    4431 scope.go:122] "RemoveContainer" containerID="ba6ef5f60b6bb55ed6ac8f7461c676a3d09b5d28692464a81de20dd1c2629baa"
Nov 03 23:00:22.968006 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:22.967983    4431 scope.go:122] "RemoveContainer" containerID="b28828735935672e686447cafd01882daefadfbcdba667ebeff32f5b0f4f94fd"
Nov 03 23:00:22.979984 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:22.979952    4431 scope.go:122] "RemoveContainer" containerID="b28828735935672e686447cafd01882daefadfbcdba667ebeff32f5b0f4f94fd"
Nov 03 23:00:22.980408 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:00:22.980380    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b28828735935672e686447cafd01882daefadfbcdba667ebeff32f5b0f4f94fd\": container with ID starting with b28828735935672e686447cafd01882daefadfbcdba667ebeff32f5b0f4f94fd not found: ID does not exist" containerID="b28828735935672e686447cafd01882daefadfbcdba667ebeff32f5b0f4f94fd"
Nov 03 23:00:22.980510 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:22.980423    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b28828735935672e686447cafd01882daefadfbcdba667ebeff32f5b0f4f94fd"} err="failed to get container status \"b28828735935672e686447cafd01882daefadfbcdba667ebeff32f5b0f4f94fd\": rpc error: code = NotFound desc = could not find container \"b28828735935672e686447cafd01882daefadfbcdba667ebeff32f5b0f4f94fd\": container with ID starting with b28828735935672e686447cafd01882daefadfbcdba667ebeff32f5b0f4f94fd not found: ID does not exist"
Nov 03 23:00:23.017674 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:23.017642    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-ff80c0-29895780-srjhc"]
Nov 03 23:00:23.022677 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:23.022494    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-ff80c0-29895780-srjhc"]
Nov 03 23:00:23.058238 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:23.057885    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-746ccb70-wxzpp" podStartSLOduration=2.057865315 podStartE2EDuration="2.057865315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:00:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:00:23.05701926 +0000 UTC m=+2847.082875126" watchObservedRunningTime="2026-11-03 23:00:23.057865315 +0000 UTC m=+2847.083721177"
Nov 03 23:00:23.095316 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:23.095289    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895780-dbslb"]
Nov 03 23:00:23.101953 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:23.101929    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895780-dbslb"]
Nov 03 23:00:23.134575 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:23.134545    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3920cb-29895780-bhr6s"]
Nov 03 23:00:23.139150 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:23.139124    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3920cb-29895780-bhr6s"]
Nov 03 23:00:23.167274 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:23.167226    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm"
Nov 03 23:00:23.167646 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:23.167618    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm"
Nov 03 23:00:23.168711 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:23.168687    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm"
Nov 03 23:00:23.172867 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:23.172845    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d8a3d1/auth-test-model-d8a3d1-kserve-5fbc7778b9-kj9kw"]
Nov 03 23:00:23.176281 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:23.176259    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d8a3d1/auth-test-model-d8a3d1-kserve-5fbc7778b9-kj9kw"]
Nov 03 23:00:23.284929 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:23.284834    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:00:23.941898 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:23.941862    4431 generic.go:356] "Generic (PLEG): container finished" podID="5ac3bb0e-6b8e-4b71-be4d-06fb3a2135b8" containerID="5743d436771d83fb1a53146f7f3ccd9f91edd6d7e37618f8241ba0ab25411c46" exitCode=0
Nov 03 23:00:23.942382 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:23.941949    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-746ccb70-wxzpp" event={"ID":"5ac3bb0e-6b8e-4b71-be4d-06fb3a2135b8","Type":"ContainerDied","Data":"5743d436771d83fb1a53146f7f3ccd9f91edd6d7e37618f8241ba0ab25411c46"}
Nov 03 23:00:24.089677 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:24.089642    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04763bf4-4b53-41a0-a47d-72d46f1f9563" path="/var/lib/kubelet/pods/04763bf4-4b53-41a0-a47d-72d46f1f9563/volumes"
Nov 03 23:00:24.090140 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:24.090127    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1503c14c-4dab-435d-b29f-5c60d0c0acfb" path="/var/lib/kubelet/pods/1503c14c-4dab-435d-b29f-5c60d0c0acfb/volumes"
Nov 03 23:00:24.090549 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:24.090535    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c362bb2-ed25-42d4-8907-7ea5985b3df4" path="/var/lib/kubelet/pods/1c362bb2-ed25-42d4-8907-7ea5985b3df4/volumes"
Nov 03 23:00:24.091328 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:24.091313    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f01190d-0246-487f-a9af-54dab722b3e3" path="/var/lib/kubelet/pods/1f01190d-0246-487f-a9af-54dab722b3e3/volumes"
Nov 03 23:00:24.091721 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:24.091709    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f6c89a6-83ae-4e3a-b10b-6da898e4857b" path="/var/lib/kubelet/pods/9f6c89a6-83ae-4e3a-b10b-6da898e4857b/volumes"
Nov 03 23:00:24.092114 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:24.092101    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6d9effb-0fd1-4443-b1ee-82600bd004ff" path="/var/lib/kubelet/pods/b6d9effb-0fd1-4443-b1ee-82600bd004ff/volumes"
Nov 03 23:00:24.092790 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:24.092774    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="becaf054-eec7-43bb-8443-8d5052e5859a" path="/var/lib/kubelet/pods/becaf054-eec7-43bb-8443-8d5052e5859a/volumes"
Nov 03 23:00:25.031353 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:25.030645    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:00:25.148752 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:25.148727    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-746ccb70-wxzpp"
Nov 03 23:00:25.217437 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:25.217344    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ac3bb0e-6b8e-4b71-be4d-06fb3a2135b8-kube-api-access-r5tk8\" (UniqueName: \"kubernetes.io/projected/5ac3bb0e-6b8e-4b71-be4d-06fb3a2135b8-kube-api-access-r5tk8\") pod \"5ac3bb0e-6b8e-4b71-be4d-06fb3a2135b8\" (UID: \"5ac3bb0e-6b8e-4b71-be4d-06fb3a2135b8\") "
Nov 03 23:00:25.217621 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:25.217441    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ac3bb0e-6b8e-4b71-be4d-06fb3a2135b8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ac3bb0e-6b8e-4b71-be4d-06fb3a2135b8-maas-api-openshift-ca\") pod \"5ac3bb0e-6b8e-4b71-be4d-06fb3a2135b8\" (UID: \"5ac3bb0e-6b8e-4b71-be4d-06fb3a2135b8\") "
Nov 03 23:00:25.217690 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:25.217648    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ac3bb0e-6b8e-4b71-be4d-06fb3a2135b8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ac3bb0e-6b8e-4b71-be4d-06fb3a2135b8-maas-api-xks-ca\") pod \"5ac3bb0e-6b8e-4b71-be4d-06fb3a2135b8\" (UID: \"5ac3bb0e-6b8e-4b71-be4d-06fb3a2135b8\") "
Nov 03 23:00:25.217842 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:25.217815    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ac3bb0e-6b8e-4b71-be4d-06fb3a2135b8-maas-api-openshift-ca" pod "5ac3bb0e-6b8e-4b71-be4d-06fb3a2135b8" (UID: "5ac3bb0e-6b8e-4b71-be4d-06fb3a2135b8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:00:25.218233 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:25.218187    4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ac3bb0e-6b8e-4b71-be4d-06fb3a2135b8-maas-api-openshift-ca\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:00:25.219583 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:25.219555    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ac3bb0e-6b8e-4b71-be4d-06fb3a2135b8-kube-api-access-r5tk8" pod "5ac3bb0e-6b8e-4b71-be4d-06fb3a2135b8" (UID: "5ac3bb0e-6b8e-4b71-be4d-06fb3a2135b8"). InnerVolumeSpecName "kube-api-access-r5tk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:00:25.219661 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:25.219639    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ac3bb0e-6b8e-4b71-be4d-06fb3a2135b8-maas-api-xks-ca" pod "5ac3bb0e-6b8e-4b71-be4d-06fb3a2135b8" (UID: "5ac3bb0e-6b8e-4b71-be4d-06fb3a2135b8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:00:25.319234 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:25.319193    4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ac3bb0e-6b8e-4b71-be4d-06fb3a2135b8-maas-api-xks-ca\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:00:25.319400 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:25.319261    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5tk8\" (UniqueName: \"kubernetes.io/projected/5ac3bb0e-6b8e-4b71-be4d-06fb3a2135b8-kube-api-access-r5tk8\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:00:25.961573 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:25.961543    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-746ccb70-wxzpp"
Nov 03 23:00:25.961573 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:25.961556    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-746ccb70-wxzpp" event={"ID":"5ac3bb0e-6b8e-4b71-be4d-06fb3a2135b8","Type":"ContainerDied","Data":"63ee348043137cb842461279324a180d64adf7af6efa2386c675e80b1abe94ae"}
Nov 03 23:00:25.961808 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:25.961643    4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63ee348043137cb842461279324a180d64adf7af6efa2386c675e80b1abe94ae"
Nov 03 23:00:26.842905 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:26.842865    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:00:27.918094 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:27.918062    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:00:28.065070 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:28.065037    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:00:28.898064 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:28.898021    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:00:30.552863 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:30.552823    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:00:31.260448 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:00:31.260410    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 23:00:32.525292 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:32.525256    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:00:34.208272 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:34.208207    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:00:35.632986 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:35.632952    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4a01f9-7854647674-2mm7m"]
Nov 03 23:00:35.633666 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:35.633647    4431 cpu_manager.go:414] "containerMap: removing container" podUID="9f6c89a6-83ae-4e3a-b10b-6da898e4857b" containerName="cleanup"
Nov 03 23:00:35.633666 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:35.633666    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f6c89a6-83ae-4e3a-b10b-6da898e4857b" containerName="cleanup"
Nov 03 23:00:35.633814 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:35.633681    4431 cpu_manager.go:414] "containerMap: removing container" podUID="b6d9effb-0fd1-4443-b1ee-82600bd004ff" containerName="cleanup"
Nov 03 23:00:35.633814 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:35.633687    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6d9effb-0fd1-4443-b1ee-82600bd004ff" containerName="cleanup"
Nov 03 23:00:35.633814 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:35.633701    4431 cpu_manager.go:414] "containerMap: removing container" podUID="9f6c89a6-83ae-4e3a-b10b-6da898e4857b" containerName="cleanup"
Nov 03 23:00:35.633814 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:35.633705    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f6c89a6-83ae-4e3a-b10b-6da898e4857b" containerName="cleanup"
Nov 03 23:00:35.633814 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:35.633713    4431 cpu_manager.go:414] "containerMap: removing container" podUID="1f01190d-0246-487f-a9af-54dab722b3e3" containerName="cleanup"
Nov 03 23:00:35.633814 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:35.633718    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f01190d-0246-487f-a9af-54dab722b3e3" containerName="cleanup"
Nov 03 23:00:35.633814 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:35.633725    4431 cpu_manager.go:414] "containerMap: removing container" podUID="b6d9effb-0fd1-4443-b1ee-82600bd004ff" containerName="cleanup"
Nov 03 23:00:35.633814 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:35.633729    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6d9effb-0fd1-4443-b1ee-82600bd004ff" containerName="cleanup"
Nov 03 23:00:35.633814 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:35.633735    4431 cpu_manager.go:414] "containerMap: removing container" podUID="04763bf4-4b53-41a0-a47d-72d46f1f9563" containerName="cleanup"
Nov 03 23:00:35.633814 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:35.633739    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04763bf4-4b53-41a0-a47d-72d46f1f9563" containerName="cleanup"
Nov 03 23:00:35.633814 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:35.633746    4431 cpu_manager.go:414] "containerMap: removing container" podUID="b6d9effb-0fd1-4443-b1ee-82600bd004ff" containerName="cleanup"
Nov 03 23:00:35.633814 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:35.633750    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6d9effb-0fd1-4443-b1ee-82600bd004ff" containerName="cleanup"
Nov 03 23:00:35.633814 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:35.633756    4431 cpu_manager.go:414] "containerMap: removing container" podUID="becaf054-eec7-43bb-8443-8d5052e5859a" containerName="cleanup"
Nov 03 23:00:35.633814 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:35.633761    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="becaf054-eec7-43bb-8443-8d5052e5859a" containerName="cleanup"
Nov 03 23:00:35.633814 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:35.633774    4431 cpu_manager.go:414] "containerMap: removing container" podUID="04763bf4-4b53-41a0-a47d-72d46f1f9563" containerName="cleanup"
Nov 03 23:00:35.633814 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:35.633779    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04763bf4-4b53-41a0-a47d-72d46f1f9563" containerName="cleanup"
Nov 03 23:00:35.633814 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:35.633785    4431 cpu_manager.go:414] "containerMap: removing container" podUID="9f6c89a6-83ae-4e3a-b10b-6da898e4857b" containerName="cleanup"
Nov 03 23:00:35.633814 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:35.633789    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f6c89a6-83ae-4e3a-b10b-6da898e4857b" containerName="cleanup"
Nov 03 23:00:35.633814 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:35.633814    4431 cpu_manager.go:414] "containerMap: removing container" podUID="1f01190d-0246-487f-a9af-54dab722b3e3" containerName="cleanup"
Nov 03 23:00:35.633814 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:35.633819    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f01190d-0246-487f-a9af-54dab722b3e3" containerName="cleanup"
Nov 03 23:00:35.633814 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:35.633825    4431 cpu_manager.go:414] "containerMap: removing container" podUID="1c362bb2-ed25-42d4-8907-7ea5985b3df4" containerName="cleanup"
Nov 03 23:00:35.634912 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:35.633830    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c362bb2-ed25-42d4-8907-7ea5985b3df4" containerName="cleanup"
Nov 03 23:00:35.634912 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:35.633837    4431 cpu_manager.go:414] "containerMap: removing container" podUID="1c362bb2-ed25-42d4-8907-7ea5985b3df4" containerName="cleanup"
Nov 03 23:00:35.634912 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:35.633841    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c362bb2-ed25-42d4-8907-7ea5985b3df4" containerName="cleanup"
Nov 03 23:00:35.634912 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:35.633848    4431 cpu_manager.go:414] "containerMap: removing container" podUID="becaf054-eec7-43bb-8443-8d5052e5859a" containerName="cleanup"
Nov 03 23:00:35.634912 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:35.633852    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="becaf054-eec7-43bb-8443-8d5052e5859a" containerName="cleanup"
Nov 03 23:00:35.634912 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:35.633857    4431 cpu_manager.go:414] "containerMap: removing container" podUID="1503c14c-4dab-435d-b29f-5c60d0c0acfb" containerName="main"
Nov 03 23:00:35.634912 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:35.633862    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1503c14c-4dab-435d-b29f-5c60d0c0acfb" containerName="main"
Nov 03 23:00:35.634912 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:35.633871    4431 cpu_manager.go:414] "containerMap: removing container" podUID="1c362bb2-ed25-42d4-8907-7ea5985b3df4" containerName="cleanup"
Nov 03 23:00:35.634912 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:35.633878    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c362bb2-ed25-42d4-8907-7ea5985b3df4" containerName="cleanup"
Nov 03 23:00:35.634912 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:35.633886    4431 cpu_manager.go:414] "containerMap: removing container" podUID="5ac3bb0e-6b8e-4b71-be4d-06fb3a2135b8" containerName="revoke-keys"
Nov 03 23:00:35.634912 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:35.633890    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ac3bb0e-6b8e-4b71-be4d-06fb3a2135b8" containerName="revoke-keys"
Nov 03 23:00:35.634912 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:35.633974    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f6c89a6-83ae-4e3a-b10b-6da898e4857b" containerName="cleanup"
Nov 03 23:00:35.634912 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:35.633984    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6d9effb-0fd1-4443-b1ee-82600bd004ff" containerName="cleanup"
Nov 03 23:00:35.634912 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:35.633993    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f6c89a6-83ae-4e3a-b10b-6da898e4857b" containerName="cleanup"
Nov 03 23:00:35.634912 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:35.633998    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f01190d-0246-487f-a9af-54dab722b3e3" containerName="cleanup"
Nov 03 23:00:35.634912 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:35.634003    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6d9effb-0fd1-4443-b1ee-82600bd004ff" containerName="cleanup"
Nov 03 23:00:35.634912 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:35.634009    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="04763bf4-4b53-41a0-a47d-72d46f1f9563" containerName="cleanup"
Nov 03 23:00:35.634912 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:35.634014    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6d9effb-0fd1-4443-b1ee-82600bd004ff" containerName="cleanup"
Nov 03 23:00:35.634912 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:35.634019    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="becaf054-eec7-43bb-8443-8d5052e5859a" containerName="cleanup"
Nov 03 23:00:35.634912 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:35.634025    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="04763bf4-4b53-41a0-a47d-72d46f1f9563" containerName="cleanup"
Nov 03 23:00:35.634912 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:35.634032    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f6c89a6-83ae-4e3a-b10b-6da898e4857b" containerName="cleanup"
Nov 03 23:00:35.634912 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:35.634040    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f01190d-0246-487f-a9af-54dab722b3e3" containerName="cleanup"
Nov 03 23:00:35.634912 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:35.634046    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c362bb2-ed25-42d4-8907-7ea5985b3df4" containerName="cleanup"
Nov 03 23:00:35.634912 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:35.634051    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="04763bf4-4b53-41a0-a47d-72d46f1f9563" containerName="cleanup"
Nov 03 23:00:35.634912 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:35.634057    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c362bb2-ed25-42d4-8907-7ea5985b3df4" containerName="cleanup"
Nov 03 23:00:35.634912 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:35.634063    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="291a5765-87a6-46b2-88f4-1d4cd9056fab" containerName="cleanup"
Nov 03 23:00:35.634912 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:35.634070    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="becaf054-eec7-43bb-8443-8d5052e5859a" containerName="cleanup"
Nov 03 23:00:35.634912 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:35.634074    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="1503c14c-4dab-435d-b29f-5c60d0c0acfb" containerName="main"
Nov 03 23:00:35.634912 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:35.634081    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ac3bb0e-6b8e-4b71-be4d-06fb3a2135b8" containerName="revoke-keys"
Nov 03 23:00:35.639056 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:35.639037    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4a01f9-7854647674-2mm7m"
Nov 03 23:00:35.672838 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:35.672808    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-4a01f9-serving-cert\""
Nov 03 23:00:35.682758 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:35.682724    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4a01f9-7854647674-2mm7m"]
Nov 03 23:00:35.763242 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:35.763193    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whb28\" (UniqueName: \"kubernetes.io/projected/2b711521-cc8c-499c-90ae-6952401b15b2-kube-api-access-whb28\") pod \"maas-api-e2e-ipp-b-4a01f9-7854647674-2mm7m\" (UID: \"2b711521-cc8c-499c-90ae-6952401b15b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4a01f9-7854647674-2mm7m"
Nov 03 23:00:35.763458 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:35.763364    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b711521-cc8c-499c-90ae-6952401b15b2-metrics-certs\") pod \"maas-api-e2e-ipp-b-4a01f9-7854647674-2mm7m\" (UID: \"2b711521-cc8c-499c-90ae-6952401b15b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4a01f9-7854647674-2mm7m"
Nov 03 23:00:35.763458 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:35.763452    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b711521-cc8c-499c-90ae-6952401b15b2-maas-api-tls\") pod \"maas-api-e2e-ipp-b-4a01f9-7854647674-2mm7m\" (UID: \"2b711521-cc8c-499c-90ae-6952401b15b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4a01f9-7854647674-2mm7m"
Nov 03 23:00:35.763580 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:35.763516    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b711521-cc8c-499c-90ae-6952401b15b2-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-4a01f9-7854647674-2mm7m\" (UID: \"2b711521-cc8c-499c-90ae-6952401b15b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4a01f9-7854647674-2mm7m"
Nov 03 23:00:35.765229 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:35.765193    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:00:35.864507 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:35.864475    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whb28\" (UniqueName: \"kubernetes.io/projected/2b711521-cc8c-499c-90ae-6952401b15b2-kube-api-access-whb28\") pod \"maas-api-e2e-ipp-b-4a01f9-7854647674-2mm7m\" (UID: \"2b711521-cc8c-499c-90ae-6952401b15b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4a01f9-7854647674-2mm7m"
Nov 03 23:00:35.864689 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:35.864562    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b711521-cc8c-499c-90ae-6952401b15b2-metrics-certs\") pod \"maas-api-e2e-ipp-b-4a01f9-7854647674-2mm7m\" (UID: \"2b711521-cc8c-499c-90ae-6952401b15b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4a01f9-7854647674-2mm7m"
Nov 03 23:00:35.864689 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:35.864602    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b711521-cc8c-499c-90ae-6952401b15b2-maas-api-tls\") pod \"maas-api-e2e-ipp-b-4a01f9-7854647674-2mm7m\" (UID: \"2b711521-cc8c-499c-90ae-6952401b15b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4a01f9-7854647674-2mm7m"
Nov 03 23:00:35.864689 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:35.864642    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b711521-cc8c-499c-90ae-6952401b15b2-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-4a01f9-7854647674-2mm7m\" (UID: \"2b711521-cc8c-499c-90ae-6952401b15b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4a01f9-7854647674-2mm7m"
Nov 03 23:00:35.865369 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:35.865342    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b711521-cc8c-499c-90ae-6952401b15b2-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-4a01f9-7854647674-2mm7m\" (UID: \"2b711521-cc8c-499c-90ae-6952401b15b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4a01f9-7854647674-2mm7m"
Nov 03 23:00:35.867037 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:35.867014    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b711521-cc8c-499c-90ae-6952401b15b2-maas-api-tls\") pod \"maas-api-e2e-ipp-b-4a01f9-7854647674-2mm7m\" (UID: \"2b711521-cc8c-499c-90ae-6952401b15b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4a01f9-7854647674-2mm7m"
Nov 03 23:00:35.867190 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:35.867166    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b711521-cc8c-499c-90ae-6952401b15b2-metrics-certs\") pod \"maas-api-e2e-ipp-b-4a01f9-7854647674-2mm7m\" (UID: \"2b711521-cc8c-499c-90ae-6952401b15b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4a01f9-7854647674-2mm7m"
Nov 03 23:00:35.887241 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:35.887167    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whb28\" (UniqueName: \"kubernetes.io/projected/2b711521-cc8c-499c-90ae-6952401b15b2-kube-api-access-whb28\") pod \"maas-api-e2e-ipp-b-4a01f9-7854647674-2mm7m\" (UID: \"2b711521-cc8c-499c-90ae-6952401b15b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4a01f9-7854647674-2mm7m"
Nov 03 23:00:35.907750 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:35.907679    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:00:35.955893 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:35.955858    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4a01f9-7854647674-2mm7m"
Nov 03 23:00:36.050904 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:36.050875    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:00:36.262762 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:36.262736    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4a01f9-7854647674-2mm7m"]
Nov 03 23:00:36.264401 ip-10-0-139-73 kubenswrapper[4431]: W1103 23:00:36.264360    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b711521_cc8c_499c_90ae_6952401b15b2.slice/crio-4356bed9dcc96946ca74c5639529070ba5c0a5cba9287d3da25009a85d4d9b19 WatchSource:0}: Error finding container 4356bed9dcc96946ca74c5639529070ba5c0a5cba9287d3da25009a85d4d9b19: Status 404 returned error can't find the container with id 4356bed9dcc96946ca74c5639529070ba5c0a5cba9287d3da25009a85d4d9b19
Nov 03 23:00:37.047917 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:37.047819    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4a01f9-7854647674-2mm7m" event={"ID":"2b711521-cc8c-499c-90ae-6952401b15b2","Type":"ContainerStarted","Data":"2ac29e4812806c4e3d82af790d1a9469105f3640ab342120b4dd5ba9f6ac00ea"}
Nov 03 23:00:37.047917 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:37.047885    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4a01f9-7854647674-2mm7m" event={"ID":"2b711521-cc8c-499c-90ae-6952401b15b2","Type":"ContainerStarted","Data":"4356bed9dcc96946ca74c5639529070ba5c0a5cba9287d3da25009a85d4d9b19"}
Nov 03 23:00:37.047917 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:37.047914    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4a01f9-7854647674-2mm7m"
Nov 03 23:00:37.090566 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:37.090504    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4a01f9-7854647674-2mm7m" podStartSLOduration=1.661559601 podStartE2EDuration="2.090488797s" totalImagesPullingTime="428.929196ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:00:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:00:37.088146986 +0000 UTC m=+2861.114002848" watchObservedRunningTime="2026-11-03 23:00:37.090488797 +0000 UTC m=+2861.116344658"
Nov 03 23:00:37.322446 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:37.322327    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:00:38.479380 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:38.479341    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:00:38.630752 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:38.630719    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:00:38.787332 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:38.787256    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:00:38.932038 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:38.931989    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:00:39.170090 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:39.169995    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:00:39.312288 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:39.312246    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:00:39.466563 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:39.466463    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:00:39.766867 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:39.766754    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:00:41.007948 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:41.007913    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:00:42.255060 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:42.255009    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:00:42.692140 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:42.692074    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:00:43.068622 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:43.068546    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4a01f9-7854647674-2mm7m"
Nov 03 23:00:44.349614 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:44.349581    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:00:46.040288 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:46.040243    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:00:46.506123 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:46.506082    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:00:46.654745 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:46.654710    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:00:46.829788 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:46.829691    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:00:47.941093 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:47.941057    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4b994-6f6f95d665-g4gs5"]
Nov 03 23:00:47.941741 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:47.941724    4431 cpu_manager.go:414] "containerMap: removing container" podUID="1f01190d-0246-487f-a9af-54dab722b3e3" containerName="cleanup"
Nov 03 23:00:47.941787 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:47.941743    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f01190d-0246-487f-a9af-54dab722b3e3" containerName="cleanup"
Nov 03 23:00:47.941787 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:47.941753    4431 cpu_manager.go:414] "containerMap: removing container" podUID="04763bf4-4b53-41a0-a47d-72d46f1f9563" containerName="cleanup"
Nov 03 23:00:47.941787 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:47.941758    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04763bf4-4b53-41a0-a47d-72d46f1f9563" containerName="cleanup"
Nov 03 23:00:47.941787 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:47.941768    4431 cpu_manager.go:414] "containerMap: removing container" podUID="becaf054-eec7-43bb-8443-8d5052e5859a" containerName="cleanup"
Nov 03 23:00:47.941787 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:47.941775    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="becaf054-eec7-43bb-8443-8d5052e5859a" containerName="cleanup"
Nov 03 23:00:47.941956 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:47.941867    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="becaf054-eec7-43bb-8443-8d5052e5859a" containerName="cleanup"
Nov 03 23:00:47.941956 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:47.941880    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f01190d-0246-487f-a9af-54dab722b3e3" containerName="cleanup"
Nov 03 23:00:47.941956 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:47.941890    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c362bb2-ed25-42d4-8907-7ea5985b3df4" containerName="cleanup"
Nov 03 23:00:47.945692 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:47.945675    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4b994-6f6f95d665-g4gs5"
Nov 03 23:00:47.966176 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:47.966147    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4b994-6f6f95d665-g4gs5"]
Nov 03 23:00:47.973474 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:47.973441    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-a4b994-serving-cert\""
Nov 03 23:00:48.071345 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:48.071314    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/309cd370-a7b9-4a9d-84ac-e4e122dec456-metrics-certs\") pod \"maas-api-e2e-ipp-a-a4b994-6f6f95d665-g4gs5\" (UID: \"309cd370-a7b9-4a9d-84ac-e4e122dec456\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4b994-6f6f95d665-g4gs5"
Nov 03 23:00:48.071553 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:48.071378    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/309cd370-a7b9-4a9d-84ac-e4e122dec456-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a4b994-6f6f95d665-g4gs5\" (UID: \"309cd370-a7b9-4a9d-84ac-e4e122dec456\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4b994-6f6f95d665-g4gs5"
Nov 03 23:00:48.071553 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:48.071485    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/309cd370-a7b9-4a9d-84ac-e4e122dec456-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a4b994-6f6f95d665-g4gs5\" (UID: \"309cd370-a7b9-4a9d-84ac-e4e122dec456\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4b994-6f6f95d665-g4gs5"
Nov 03 23:00:48.071553 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:48.071545    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhdsp\" (UniqueName: \"kubernetes.io/projected/309cd370-a7b9-4a9d-84ac-e4e122dec456-kube-api-access-qhdsp\") pod \"maas-api-e2e-ipp-a-a4b994-6f6f95d665-g4gs5\" (UID: \"309cd370-a7b9-4a9d-84ac-e4e122dec456\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4b994-6f6f95d665-g4gs5"
Nov 03 23:00:48.084810 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:48.084751    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:00:48.172979 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:48.172944    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/309cd370-a7b9-4a9d-84ac-e4e122dec456-metrics-certs\") pod \"maas-api-e2e-ipp-a-a4b994-6f6f95d665-g4gs5\" (UID: \"309cd370-a7b9-4a9d-84ac-e4e122dec456\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4b994-6f6f95d665-g4gs5"
Nov 03 23:00:48.173167 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:48.173039    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/309cd370-a7b9-4a9d-84ac-e4e122dec456-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a4b994-6f6f95d665-g4gs5\" (UID: \"309cd370-a7b9-4a9d-84ac-e4e122dec456\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4b994-6f6f95d665-g4gs5"
Nov 03 23:00:48.173167 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:48.173110    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/309cd370-a7b9-4a9d-84ac-e4e122dec456-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a4b994-6f6f95d665-g4gs5\" (UID: \"309cd370-a7b9-4a9d-84ac-e4e122dec456\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4b994-6f6f95d665-g4gs5"
Nov 03 23:00:48.173309 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:48.173176    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhdsp\" (UniqueName: \"kubernetes.io/projected/309cd370-a7b9-4a9d-84ac-e4e122dec456-kube-api-access-qhdsp\") pod \"maas-api-e2e-ipp-a-a4b994-6f6f95d665-g4gs5\" (UID: \"309cd370-a7b9-4a9d-84ac-e4e122dec456\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4b994-6f6f95d665-g4gs5"
Nov 03 23:00:48.173309 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:00:48.173192    4431 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4b994-serving-cert: secret "maas-api-e2e-ipp-a-a4b994-serving-cert" not found
Nov 03 23:00:48.173421 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:00:48.173307    4431 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/309cd370-a7b9-4a9d-84ac-e4e122dec456-maas-api-tls podName:309cd370-a7b9-4a9d-84ac-e4e122dec456 nodeName:}" failed. No retries permitted until 2026-11-03 23:00:48.673282284 +0000 UTC m=+2872.699138138 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/309cd370-a7b9-4a9d-84ac-e4e122dec456-maas-api-tls") pod "maas-api-e2e-ipp-a-a4b994-6f6f95d665-g4gs5" (UID: "309cd370-a7b9-4a9d-84ac-e4e122dec456") : secret "maas-api-e2e-ipp-a-a4b994-serving-cert" not found
Nov 03 23:00:48.173855 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:48.173833    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/309cd370-a7b9-4a9d-84ac-e4e122dec456-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a4b994-6f6f95d665-g4gs5\" (UID: \"309cd370-a7b9-4a9d-84ac-e4e122dec456\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4b994-6f6f95d665-g4gs5"
Nov 03 23:00:48.175827 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:48.175808    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/309cd370-a7b9-4a9d-84ac-e4e122dec456-metrics-certs\") pod \"maas-api-e2e-ipp-a-a4b994-6f6f95d665-g4gs5\" (UID: \"309cd370-a7b9-4a9d-84ac-e4e122dec456\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4b994-6f6f95d665-g4gs5"
Nov 03 23:00:48.201523 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:48.201453    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhdsp\" (UniqueName: \"kubernetes.io/projected/309cd370-a7b9-4a9d-84ac-e4e122dec456-kube-api-access-qhdsp\") pod \"maas-api-e2e-ipp-a-a4b994-6f6f95d665-g4gs5\" (UID: \"309cd370-a7b9-4a9d-84ac-e4e122dec456\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4b994-6f6f95d665-g4gs5"
Nov 03 23:00:48.232461 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:48.232402    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:00:48.377819 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:48.377776    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:00:48.679668 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:48.679631    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/309cd370-a7b9-4a9d-84ac-e4e122dec456-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a4b994-6f6f95d665-g4gs5\" (UID: \"309cd370-a7b9-4a9d-84ac-e4e122dec456\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4b994-6f6f95d665-g4gs5"
Nov 03 23:00:48.682260 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:48.682231    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/309cd370-a7b9-4a9d-84ac-e4e122dec456-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a4b994-6f6f95d665-g4gs5\" (UID: \"309cd370-a7b9-4a9d-84ac-e4e122dec456\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4b994-6f6f95d665-g4gs5"
Nov 03 23:00:48.860419 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:48.860384    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4b994-6f6f95d665-g4gs5"
Nov 03 23:00:49.192127 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:49.192095    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4b994-6f6f95d665-g4gs5"]
Nov 03 23:00:49.194434 ip-10-0-139-73 kubenswrapper[4431]: W1103 23:00:49.194407    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod309cd370_a7b9_4a9d_84ac_e4e122dec456.slice/crio-8a83d76333d9fd12b282b823cb48dc20182fa4859e1031b91dca0acf311cc5f0 WatchSource:0}: Error finding container 8a83d76333d9fd12b282b823cb48dc20182fa4859e1031b91dca0acf311cc5f0: Status 404 returned error can't find the container with id 8a83d76333d9fd12b282b823cb48dc20182fa4859e1031b91dca0acf311cc5f0
Nov 03 23:00:49.264266 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:49.262466    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4b994-6f6f95d665-g4gs5" event={"ID":"309cd370-a7b9-4a9d-84ac-e4e122dec456","Type":"ContainerStarted","Data":"8a83d76333d9fd12b282b823cb48dc20182fa4859e1031b91dca0acf311cc5f0"}
Nov 03 23:00:50.022890 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:50.022854    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:00:50.167193 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:50.167158    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:00:50.275994 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:50.275893    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4b994-6f6f95d665-g4gs5" event={"ID":"309cd370-a7b9-4a9d-84ac-e4e122dec456","Type":"ContainerStarted","Data":"ba08c1ded0a276322b565bac9e2134f4bfbaa8840593d52c17fabfc294924224"}
Nov 03 23:00:50.276827 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:50.276766    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4b994-6f6f95d665-g4gs5"
Nov 03 23:00:50.319437 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:50.319381    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4b994-6f6f95d665-g4gs5" podStartSLOduration=2.953293952 podStartE2EDuration="3.319366948s" totalImagesPullingTime="366.072996ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:00:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:00:50.317045464 +0000 UTC m=+2874.342901326" watchObservedRunningTime="2026-11-03 23:00:50.319366948 +0000 UTC m=+2874.345222810"
Nov 03 23:00:51.580472 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:51.580436    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:00:53.613742 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:53.613710    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:00:54.805669 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:54.805630    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:00:54.945124 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:54.945091    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:00:55.123194 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:55.123094    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:00:55.308970 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:55.308928    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:00:56.293653 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:56.293620    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4b994-6f6f95d665-g4gs5"
Nov 03 23:00:56.720440 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:56.720402    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:00:58.286169 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:00:58.286122    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:01:00.002291 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:00.002245    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:01:01.263300 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:01:01.263256    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 23:01:01.280849 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:01.280811    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:01:01.451324 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:01.451259    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:01:01.594028 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:01.593922    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:01:01.850921 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:01.850840    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:01:01.995677 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:01.995641    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:01:02.149093 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:02.149005    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:01:02.470718 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:02.470680    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:01:03.636146 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:03.636112    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:01:05.220532 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:05.220493    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:01:06.778491 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:06.778454    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:01:08.496166 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:08.496124    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:01:08.904503 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:08.904410    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:01:09.554374 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:09.554334    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:01:09.756568 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:09.756532    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:01:09.880488 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:09.880353    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2"]
Nov 03 23:01:09.880670 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:09.880621    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2" podUID="681f7328-e1f5-4544-876d-410f30ee83e5" containerName="istio-proxy" containerID="cri-o://eb013844454b316fbcb6950f42574565ea2b70fa136c40a95058fb2bb79aeb47" gracePeriod=30
Nov 03 23:01:09.912104 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:09.912073    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:01:10.172825 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:10.172760    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:01:10.390439 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:10.390406    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:01:12.387204 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:12.387141    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:01:14.030072 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:14.030034    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:01:15.163065 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:15.163044    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2"
Nov 03 23:01:15.252774 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:15.252734    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/681f7328-e1f5-4544-876d-410f30ee83e5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/681f7328-e1f5-4544-876d-410f30ee83e5-istio-data\") pod \"681f7328-e1f5-4544-876d-410f30ee83e5\" (UID: \"681f7328-e1f5-4544-876d-410f30ee83e5\") "
Nov 03 23:01:15.252994 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:15.252928    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/681f7328-e1f5-4544-876d-410f30ee83e5-kube-api-access-vtnf5\" (UniqueName: \"kubernetes.io/projected/681f7328-e1f5-4544-876d-410f30ee83e5-kube-api-access-vtnf5\") pod \"681f7328-e1f5-4544-876d-410f30ee83e5\" (UID: \"681f7328-e1f5-4544-876d-410f30ee83e5\") "
Nov 03 23:01:15.252994 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:15.252968    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/681f7328-e1f5-4544-876d-410f30ee83e5-istio-data" pod "681f7328-e1f5-4544-876d-410f30ee83e5" (UID: "681f7328-e1f5-4544-876d-410f30ee83e5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:01:15.253147 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:15.253030    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/681f7328-e1f5-4544-876d-410f30ee83e5-istio-token\" (UniqueName: \"kubernetes.io/projected/681f7328-e1f5-4544-876d-410f30ee83e5-istio-token\") pod \"681f7328-e1f5-4544-876d-410f30ee83e5\" (UID: \"681f7328-e1f5-4544-876d-410f30ee83e5\") "
Nov 03 23:01:15.253147 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:15.253066    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/681f7328-e1f5-4544-876d-410f30ee83e5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/681f7328-e1f5-4544-876d-410f30ee83e5-workload-certs\") pod \"681f7328-e1f5-4544-876d-410f30ee83e5\" (UID: \"681f7328-e1f5-4544-876d-410f30ee83e5\") "
Nov 03 23:01:15.253147 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:15.253118    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/681f7328-e1f5-4544-876d-410f30ee83e5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/681f7328-e1f5-4544-876d-410f30ee83e5-istiod-ca-cert\") pod \"681f7328-e1f5-4544-876d-410f30ee83e5\" (UID: \"681f7328-e1f5-4544-876d-410f30ee83e5\") "
Nov 03 23:01:15.253348 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:15.253192    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/681f7328-e1f5-4544-876d-410f30ee83e5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/681f7328-e1f5-4544-876d-410f30ee83e5-istio-podinfo\") pod \"681f7328-e1f5-4544-876d-410f30ee83e5\" (UID: \"681f7328-e1f5-4544-876d-410f30ee83e5\") "
Nov 03 23:01:15.253348 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:15.253261    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/681f7328-e1f5-4544-876d-410f30ee83e5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/681f7328-e1f5-4544-876d-410f30ee83e5-workload-socket\") pod \"681f7328-e1f5-4544-876d-410f30ee83e5\" (UID: \"681f7328-e1f5-4544-876d-410f30ee83e5\") "
Nov 03 23:01:15.253348 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:15.253302    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/681f7328-e1f5-4544-876d-410f30ee83e5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/681f7328-e1f5-4544-876d-410f30ee83e5-istio-envoy\") pod \"681f7328-e1f5-4544-876d-410f30ee83e5\" (UID: \"681f7328-e1f5-4544-876d-410f30ee83e5\") "
Nov 03 23:01:15.253533 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:15.253363    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/681f7328-e1f5-4544-876d-410f30ee83e5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/681f7328-e1f5-4544-876d-410f30ee83e5-credential-socket\") pod \"681f7328-e1f5-4544-876d-410f30ee83e5\" (UID: \"681f7328-e1f5-4544-876d-410f30ee83e5\") "
Nov 03 23:01:15.253604 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:15.253538    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/681f7328-e1f5-4544-876d-410f30ee83e5-workload-socket" pod "681f7328-e1f5-4544-876d-410f30ee83e5" (UID: "681f7328-e1f5-4544-876d-410f30ee83e5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:01:15.253879 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:15.253853    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/681f7328-e1f5-4544-876d-410f30ee83e5-istiod-ca-cert" pod "681f7328-e1f5-4544-876d-410f30ee83e5" (UID: "681f7328-e1f5-4544-876d-410f30ee83e5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:01:15.254168 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:15.254144    4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/681f7328-e1f5-4544-876d-410f30ee83e5-istio-data\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:01:15.254369 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:15.254354    4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/681f7328-e1f5-4544-876d-410f30ee83e5-istiod-ca-cert\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:01:15.254535 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:15.254521    4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/681f7328-e1f5-4544-876d-410f30ee83e5-workload-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:01:15.254654 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:15.254245    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/681f7328-e1f5-4544-876d-410f30ee83e5-credential-socket" pod "681f7328-e1f5-4544-876d-410f30ee83e5" (UID: "681f7328-e1f5-4544-876d-410f30ee83e5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:01:15.254654 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:15.254399    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/681f7328-e1f5-4544-876d-410f30ee83e5-workload-certs" pod "681f7328-e1f5-4544-876d-410f30ee83e5" (UID: "681f7328-e1f5-4544-876d-410f30ee83e5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:01:15.255669 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:15.255644    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/681f7328-e1f5-4544-876d-410f30ee83e5-istio-token" pod "681f7328-e1f5-4544-876d-410f30ee83e5" (UID: "681f7328-e1f5-4544-876d-410f30ee83e5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:01:15.255785 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:15.255760    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/681f7328-e1f5-4544-876d-410f30ee83e5-kube-api-access-vtnf5" pod "681f7328-e1f5-4544-876d-410f30ee83e5" (UID: "681f7328-e1f5-4544-876d-410f30ee83e5"). InnerVolumeSpecName "kube-api-access-vtnf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:01:15.255994 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:15.255973    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/681f7328-e1f5-4544-876d-410f30ee83e5-istio-podinfo" pod "681f7328-e1f5-4544-876d-410f30ee83e5" (UID: "681f7328-e1f5-4544-876d-410f30ee83e5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 23:01:15.256297 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:15.256279    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/681f7328-e1f5-4544-876d-410f30ee83e5-istio-envoy" pod "681f7328-e1f5-4544-876d-410f30ee83e5" (UID: "681f7328-e1f5-4544-876d-410f30ee83e5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:01:15.356530 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:15.356499    4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/681f7328-e1f5-4544-876d-410f30ee83e5-istio-podinfo\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:01:15.356530 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:15.356525    4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/681f7328-e1f5-4544-876d-410f30ee83e5-istio-envoy\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:01:15.356530 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:15.356534    4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/681f7328-e1f5-4544-876d-410f30ee83e5-credential-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:01:15.356776 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:15.356544    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtnf5\" (UniqueName: \"kubernetes.io/projected/681f7328-e1f5-4544-876d-410f30ee83e5-kube-api-access-vtnf5\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:01:15.356776 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:15.356554    4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/681f7328-e1f5-4544-876d-410f30ee83e5-istio-token\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:01:15.356776 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:15.356562    4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/681f7328-e1f5-4544-876d-410f30ee83e5-workload-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:01:15.462854 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:15.462767    4431 generic.go:356] "Generic (PLEG): container finished" podID="681f7328-e1f5-4544-876d-410f30ee83e5" containerID="eb013844454b316fbcb6950f42574565ea2b70fa136c40a95058fb2bb79aeb47" exitCode=0
Nov 03 23:01:15.462854 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:15.462815    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2"
Nov 03 23:01:15.462854 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:15.462846    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2" event={"ID":"681f7328-e1f5-4544-876d-410f30ee83e5","Type":"ContainerDied","Data":"eb013844454b316fbcb6950f42574565ea2b70fa136c40a95058fb2bb79aeb47"}
Nov 03 23:01:15.463070 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:15.462921    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2" event={"ID":"681f7328-e1f5-4544-876d-410f30ee83e5","Type":"ContainerDied","Data":"0b79a8d0c0944b3ed0c9ce2bf75072dff1c26f569120e37528534fff53499c47"}
Nov 03 23:01:15.463070 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:15.462942    4431 scope.go:122] "RemoveContainer" containerID="eb013844454b316fbcb6950f42574565ea2b70fa136c40a95058fb2bb79aeb47"
Nov 03 23:01:15.474090 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:15.474074    4431 scope.go:122] "RemoveContainer" containerID="56cd9f39de4e5a13cba87a2ef1dcb98947e79cdbcb8bc3c4e4731bb1c4248148"
Nov 03 23:01:15.480456 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:15.480434    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:01:15.483928 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:15.483913    4431 scope.go:122] "RemoveContainer" containerID="eb013844454b316fbcb6950f42574565ea2b70fa136c40a95058fb2bb79aeb47"
Nov 03 23:01:15.484198 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:01:15.484177    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb013844454b316fbcb6950f42574565ea2b70fa136c40a95058fb2bb79aeb47\": container with ID starting with eb013844454b316fbcb6950f42574565ea2b70fa136c40a95058fb2bb79aeb47 not found: ID does not exist" containerID="eb013844454b316fbcb6950f42574565ea2b70fa136c40a95058fb2bb79aeb47"
Nov 03 23:01:15.484268 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:15.484209    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb013844454b316fbcb6950f42574565ea2b70fa136c40a95058fb2bb79aeb47"} err="failed to get container status \"eb013844454b316fbcb6950f42574565ea2b70fa136c40a95058fb2bb79aeb47\": rpc error: code = NotFound desc = could not find container \"eb013844454b316fbcb6950f42574565ea2b70fa136c40a95058fb2bb79aeb47\": container with ID starting with eb013844454b316fbcb6950f42574565ea2b70fa136c40a95058fb2bb79aeb47 not found: ID does not exist"
Nov 03 23:01:15.484268 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:15.484252    4431 scope.go:122] "RemoveContainer" containerID="56cd9f39de4e5a13cba87a2ef1dcb98947e79cdbcb8bc3c4e4731bb1c4248148"
Nov 03 23:01:15.484486 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:01:15.484470    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56cd9f39de4e5a13cba87a2ef1dcb98947e79cdbcb8bc3c4e4731bb1c4248148\": container with ID starting with 56cd9f39de4e5a13cba87a2ef1dcb98947e79cdbcb8bc3c4e4731bb1c4248148 not found: ID does not exist" containerID="56cd9f39de4e5a13cba87a2ef1dcb98947e79cdbcb8bc3c4e4731bb1c4248148"
Nov 03 23:01:15.484541 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:15.484490    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56cd9f39de4e5a13cba87a2ef1dcb98947e79cdbcb8bc3c4e4731bb1c4248148"} err="failed to get container status \"56cd9f39de4e5a13cba87a2ef1dcb98947e79cdbcb8bc3c4e4731bb1c4248148\": rpc error: code = NotFound desc = could not find container \"56cd9f39de4e5a13cba87a2ef1dcb98947e79cdbcb8bc3c4e4731bb1c4248148\": container with ID starting with 56cd9f39de4e5a13cba87a2ef1dcb98947e79cdbcb8bc3c4e4731bb1c4248148 not found: ID does not exist"
Nov 03 23:01:15.516204 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:15.516170    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2"]
Nov 03 23:01:15.527549 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:15.527520    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-4rzs2"]
Nov 03 23:01:16.095038 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:16.094999    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="681f7328-e1f5-4544-876d-410f30ee83e5" path="/var/lib/kubelet/pods/681f7328-e1f5-4544-876d-410f30ee83e5/volumes"
Nov 03 23:01:16.314808 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:16.314770    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:01:18.018376 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:18.018329    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:01:19.611545 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:19.611502    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:01:21.325299 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:21.325264    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:01:22.265857 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:22.265822    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:01:22.411407 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:22.411372    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:01:22.985600 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:22.985567    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:01:23.049713 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:23.049661    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4a01f9-57fb584d-fn5pv"]
Nov 03 23:01:23.162681 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:23.162650    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4a01f9-6df8b8cb59-l6zlr"]
Nov 03 23:01:23.332618 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:23.332522    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:01:23.476535 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:23.476503    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:01:24.634326 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:24.634292    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:01:26.200779 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:26.200741    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:01:27.260403 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:27.260371    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-a940d5/sub-test-model-a940d5-kserve-656c9f7b4c-k7tz5"]
Nov 03 23:01:27.361967 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:27.361928    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:01:27.887895 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:27.887866    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:01:28.028912 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:28.028876    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:01:28.050946 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:28.050903    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-4a01f9-57fb584d-fn5pv" podUID="d1b37d17-4abd-408b-a1dc-b89eefeb3393" containerName="payload-processing" containerID="cri-o://64554b336c7280e5b3e62d9913efc83887509eb1469a448d547d8ffe28e2f853" gracePeriod=25
Nov 03 23:01:28.163691 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:28.163659    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4a01f9-6df8b8cb59-l6zlr" podUID="e02c5146-c15c-4b43-9448-457efdde2d2c" containerName="payload-pre-processing" containerID="cri-o://b9933420b8878d6068e77328e964d06b6eef60e5250ab71c43dc4c24698d1652" gracePeriod=25
Nov 03 23:01:28.375999 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:28.375971    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4a01f9-57fb584d-fn5pv"
Nov 03 23:01:28.378583 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:28.378560    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:01:28.499399 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:28.499308    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1b37d17-4abd-408b-a1dc-b89eefeb3393-kube-api-access-wwdsz\" (UniqueName: \"kubernetes.io/projected/d1b37d17-4abd-408b-a1dc-b89eefeb3393-kube-api-access-wwdsz\") pod \"d1b37d17-4abd-408b-a1dc-b89eefeb3393\" (UID: \"d1b37d17-4abd-408b-a1dc-b89eefeb3393\") "
Nov 03 23:01:28.500143 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:28.500116    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1b37d17-4abd-408b-a1dc-b89eefeb3393-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1b37d17-4abd-408b-a1dc-b89eefeb3393-plugins-config-volume\") pod \"d1b37d17-4abd-408b-a1dc-b89eefeb3393\" (UID: \"d1b37d17-4abd-408b-a1dc-b89eefeb3393\") "
Nov 03 23:01:28.500840 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:28.500816    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1b37d17-4abd-408b-a1dc-b89eefeb3393-plugins-config-volume" pod "d1b37d17-4abd-408b-a1dc-b89eefeb3393" (UID: "d1b37d17-4abd-408b-a1dc-b89eefeb3393"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:01:28.527707 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:28.527676    4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1b37d17-4abd-408b-a1dc-b89eefeb3393-plugins-config-volume\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:01:28.530196 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:28.530149    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1b37d17-4abd-408b-a1dc-b89eefeb3393-kube-api-access-wwdsz" pod "d1b37d17-4abd-408b-a1dc-b89eefeb3393" (UID: "d1b37d17-4abd-408b-a1dc-b89eefeb3393"). InnerVolumeSpecName "kube-api-access-wwdsz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:01:28.572437 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:28.572400    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4a01f9-6df8b8cb59-l6zlr"
Nov 03 23:01:28.579697 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:28.579667    4431 generic.go:356] "Generic (PLEG): container finished" podID="e02c5146-c15c-4b43-9448-457efdde2d2c" containerID="b9933420b8878d6068e77328e964d06b6eef60e5250ab71c43dc4c24698d1652" exitCode=0
Nov 03 23:01:28.579881 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:28.579711    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4a01f9-6df8b8cb59-l6zlr"
Nov 03 23:01:28.579881 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:28.579748    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4a01f9-6df8b8cb59-l6zlr" event={"ID":"e02c5146-c15c-4b43-9448-457efdde2d2c","Type":"ContainerDied","Data":"b9933420b8878d6068e77328e964d06b6eef60e5250ab71c43dc4c24698d1652"}
Nov 03 23:01:28.579881 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:28.579813    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4a01f9-6df8b8cb59-l6zlr" event={"ID":"e02c5146-c15c-4b43-9448-457efdde2d2c","Type":"ContainerDied","Data":"b3387282691143981a5391bfd5c250e00769b512b30263b177c80a02b1aa8352"}
Nov 03 23:01:28.579881 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:28.579845    4431 scope.go:122] "RemoveContainer" containerID="b9933420b8878d6068e77328e964d06b6eef60e5250ab71c43dc4c24698d1652"
Nov 03 23:01:28.581349 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:28.581317    4431 generic.go:356] "Generic (PLEG): container finished" podID="d1b37d17-4abd-408b-a1dc-b89eefeb3393" containerID="64554b336c7280e5b3e62d9913efc83887509eb1469a448d547d8ffe28e2f853" exitCode=0
Nov 03 23:01:28.581501 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:28.581363    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4a01f9-57fb584d-fn5pv"
Nov 03 23:01:28.581501 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:28.581404    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4a01f9-57fb584d-fn5pv" event={"ID":"d1b37d17-4abd-408b-a1dc-b89eefeb3393","Type":"ContainerDied","Data":"64554b336c7280e5b3e62d9913efc83887509eb1469a448d547d8ffe28e2f853"}
Nov 03 23:01:28.581501 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:28.581449    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4a01f9-57fb584d-fn5pv" event={"ID":"d1b37d17-4abd-408b-a1dc-b89eefeb3393","Type":"ContainerDied","Data":"844cd1779d3d94b194d76c028aef99097b0fb67293914d91c0f721025d5c12f0"}
Nov 03 23:01:28.598211 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:28.598162    4431 scope.go:122] "RemoveContainer" containerID="b9933420b8878d6068e77328e964d06b6eef60e5250ab71c43dc4c24698d1652"
Nov 03 23:01:28.598663 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:01:28.598637    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9933420b8878d6068e77328e964d06b6eef60e5250ab71c43dc4c24698d1652\": container with ID starting with b9933420b8878d6068e77328e964d06b6eef60e5250ab71c43dc4c24698d1652 not found: ID does not exist" containerID="b9933420b8878d6068e77328e964d06b6eef60e5250ab71c43dc4c24698d1652"
Nov 03 23:01:28.598742 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:28.598677    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9933420b8878d6068e77328e964d06b6eef60e5250ab71c43dc4c24698d1652"} err="failed to get container status \"b9933420b8878d6068e77328e964d06b6eef60e5250ab71c43dc4c24698d1652\": rpc error: code = NotFound desc = could not find container \"b9933420b8878d6068e77328e964d06b6eef60e5250ab71c43dc4c24698d1652\": container with ID starting with b9933420b8878d6068e77328e964d06b6eef60e5250ab71c43dc4c24698d1652 not found: ID does not exist"
Nov 03 23:01:28.598742 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:28.598701    4431 scope.go:122] "RemoveContainer" containerID="64554b336c7280e5b3e62d9913efc83887509eb1469a448d547d8ffe28e2f853"
Nov 03 23:01:28.611944 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:28.611921    4431 scope.go:122] "RemoveContainer" containerID="64554b336c7280e5b3e62d9913efc83887509eb1469a448d547d8ffe28e2f853"
Nov 03 23:01:28.612307 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:01:28.612278    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64554b336c7280e5b3e62d9913efc83887509eb1469a448d547d8ffe28e2f853\": container with ID starting with 64554b336c7280e5b3e62d9913efc83887509eb1469a448d547d8ffe28e2f853 not found: ID does not exist" containerID="64554b336c7280e5b3e62d9913efc83887509eb1469a448d547d8ffe28e2f853"
Nov 03 23:01:28.612408 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:28.612311    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64554b336c7280e5b3e62d9913efc83887509eb1469a448d547d8ffe28e2f853"} err="failed to get container status \"64554b336c7280e5b3e62d9913efc83887509eb1469a448d547d8ffe28e2f853\": rpc error: code = NotFound desc = could not find container \"64554b336c7280e5b3e62d9913efc83887509eb1469a448d547d8ffe28e2f853\": container with ID starting with 64554b336c7280e5b3e62d9913efc83887509eb1469a448d547d8ffe28e2f853 not found: ID does not exist"
Nov 03 23:01:28.629403 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:28.629356    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e02c5146-c15c-4b43-9448-457efdde2d2c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e02c5146-c15c-4b43-9448-457efdde2d2c-plugins-config-volume\") pod \"e02c5146-c15c-4b43-9448-457efdde2d2c\" (UID: \"e02c5146-c15c-4b43-9448-457efdde2d2c\") "
Nov 03 23:01:28.629606 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:28.629587    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e02c5146-c15c-4b43-9448-457efdde2d2c-kube-api-access-hsld2\" (UniqueName: \"kubernetes.io/projected/e02c5146-c15c-4b43-9448-457efdde2d2c-kube-api-access-hsld2\") pod \"e02c5146-c15c-4b43-9448-457efdde2d2c\" (UID: \"e02c5146-c15c-4b43-9448-457efdde2d2c\") "
Nov 03 23:01:28.629744 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:28.629719    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e02c5146-c15c-4b43-9448-457efdde2d2c-plugins-config-volume" pod "e02c5146-c15c-4b43-9448-457efdde2d2c" (UID: "e02c5146-c15c-4b43-9448-457efdde2d2c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:01:28.630371 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:28.630332    4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e02c5146-c15c-4b43-9448-457efdde2d2c-plugins-config-volume\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:01:28.630476 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:28.630379    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwdsz\" (UniqueName: \"kubernetes.io/projected/d1b37d17-4abd-408b-a1dc-b89eefeb3393-kube-api-access-wwdsz\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:01:28.632749 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:28.632722    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e02c5146-c15c-4b43-9448-457efdde2d2c-kube-api-access-hsld2" pod "e02c5146-c15c-4b43-9448-457efdde2d2c" (UID: "e02c5146-c15c-4b43-9448-457efdde2d2c"). InnerVolumeSpecName "kube-api-access-hsld2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:01:28.637045 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:28.637014    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:01:28.682701 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:28.678346    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4a01f9-57fb584d-fn5pv"]
Nov 03 23:01:28.683150 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:28.683126    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4a01f9-57fb584d-fn5pv"]
Nov 03 23:01:28.731344 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:28.731315    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsld2\" (UniqueName: \"kubernetes.io/projected/e02c5146-c15c-4b43-9448-457efdde2d2c-kube-api-access-hsld2\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:01:28.925846 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:28.925811    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4a01f9-6df8b8cb59-l6zlr"]
Nov 03 23:01:28.931970 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:28.931936    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4a01f9-6df8b8cb59-l6zlr"]
Nov 03 23:01:29.744271 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:29.744197    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:01:30.091068 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:30.090992    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1b37d17-4abd-408b-a1dc-b89eefeb3393" path="/var/lib/kubelet/pods/d1b37d17-4abd-408b-a1dc-b89eefeb3393/volumes"
Nov 03 23:01:30.091530 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:30.091514    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e02c5146-c15c-4b43-9448-457efdde2d2c" path="/var/lib/kubelet/pods/e02c5146-c15c-4b43-9448-457efdde2d2c/volumes"
Nov 03 23:01:31.259720 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:01:31.259688    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 23:01:31.376199 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:31.373761    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:01:32.304242 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:32.304180    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:01:32.449000 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:32.448965    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:01:33.218365 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:33.218332    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:01:34.950716 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:34.950677    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:01:35.512685 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:35.512646    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:01:35.685106 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:35.685065    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:01:36.080238 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:36.080190    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:01:36.236304 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:36.236264    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:01:36.474502 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:36.474465    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:01:36.846929 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:36.846851    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:01:37.215655 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:37.215616    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:01:37.513547 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:37.513456    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:01:37.687402 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:37.687360    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:01:39.062856 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:39.062811    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:01:40.740834 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:40.740742    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:01:40.985984 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:40.985947    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:01:42.291528 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:42.291484    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-a940d5/sub-test-model-a940d5-kserve-656c9f7b4c-k7tz5" podUID="7ac36a59-81e9-4274-b40d-870331ee4c34" containerName="main" containerID="cri-o://346427fcd41cfc506c2760934837af5435cdfd89d382bea6eea0aaaabe85ccfc" gracePeriod=45
Nov 03 23:01:42.461088 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:42.461044    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:01:42.588857 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:42.588827    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-a940d5/sub-test-model-a940d5-kserve-656c9f7b4c-k7tz5"
Nov 03 23:01:42.654612 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:42.654576    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:01:42.693024 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:42.692991    4431 generic.go:356] "Generic (PLEG): container finished" podID="7ac36a59-81e9-4274-b40d-870331ee4c34" containerID="346427fcd41cfc506c2760934837af5435cdfd89d382bea6eea0aaaabe85ccfc" exitCode=0
Nov 03 23:01:42.693230 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:42.693037    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-a940d5/sub-test-model-a940d5-kserve-656c9f7b4c-k7tz5"
Nov 03 23:01:42.693230 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:42.693073    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-a940d5/sub-test-model-a940d5-kserve-656c9f7b4c-k7tz5" event={"ID":"7ac36a59-81e9-4274-b40d-870331ee4c34","Type":"ContainerDied","Data":"346427fcd41cfc506c2760934837af5435cdfd89d382bea6eea0aaaabe85ccfc"}
Nov 03 23:01:42.693230 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:42.693118    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-a940d5/sub-test-model-a940d5-kserve-656c9f7b4c-k7tz5" event={"ID":"7ac36a59-81e9-4274-b40d-870331ee4c34","Type":"ContainerDied","Data":"4871aafcb924f1b7c7bcc90ec9e664b8d9325297c30f0d67b1bbd8a603acd502"}
Nov 03 23:01:42.693230 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:42.693136    4431 scope.go:122] "RemoveContainer" containerID="346427fcd41cfc506c2760934837af5435cdfd89d382bea6eea0aaaabe85ccfc"
Nov 03 23:01:42.703084 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:42.703067    4431 scope.go:122] "RemoveContainer" containerID="346427fcd41cfc506c2760934837af5435cdfd89d382bea6eea0aaaabe85ccfc"
Nov 03 23:01:42.703471 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:01:42.703449    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"346427fcd41cfc506c2760934837af5435cdfd89d382bea6eea0aaaabe85ccfc\": container with ID starting with 346427fcd41cfc506c2760934837af5435cdfd89d382bea6eea0aaaabe85ccfc not found: ID does not exist" containerID="346427fcd41cfc506c2760934837af5435cdfd89d382bea6eea0aaaabe85ccfc"
Nov 03 23:01:42.703533 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:42.703482    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"346427fcd41cfc506c2760934837af5435cdfd89d382bea6eea0aaaabe85ccfc"} err="failed to get container status \"346427fcd41cfc506c2760934837af5435cdfd89d382bea6eea0aaaabe85ccfc\": rpc error: code = NotFound desc = could not find container \"346427fcd41cfc506c2760934837af5435cdfd89d382bea6eea0aaaabe85ccfc\": container with ID starting with 346427fcd41cfc506c2760934837af5435cdfd89d382bea6eea0aaaabe85ccfc not found: ID does not exist"
Nov 03 23:01:42.729207 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:42.729173    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ac36a59-81e9-4274-b40d-870331ee4c34-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ac36a59-81e9-4274-b40d-870331ee4c34-tmp-dir\") pod \"7ac36a59-81e9-4274-b40d-870331ee4c34\" (UID: \"7ac36a59-81e9-4274-b40d-870331ee4c34\") "
Nov 03 23:01:42.729439 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:42.729264    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ac36a59-81e9-4274-b40d-870331ee4c34-kube-api-access-45jgg\" (UniqueName: \"kubernetes.io/projected/7ac36a59-81e9-4274-b40d-870331ee4c34-kube-api-access-45jgg\") pod \"7ac36a59-81e9-4274-b40d-870331ee4c34\" (UID: \"7ac36a59-81e9-4274-b40d-870331ee4c34\") "
Nov 03 23:01:42.729439 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:42.729286    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ac36a59-81e9-4274-b40d-870331ee4c34-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ac36a59-81e9-4274-b40d-870331ee4c34-dshm\") pod \"7ac36a59-81e9-4274-b40d-870331ee4c34\" (UID: \"7ac36a59-81e9-4274-b40d-870331ee4c34\") "
Nov 03 23:01:42.729439 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:42.729364    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ac36a59-81e9-4274-b40d-870331ee4c34-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ac36a59-81e9-4274-b40d-870331ee4c34-model-cache\") pod \"7ac36a59-81e9-4274-b40d-870331ee4c34\" (UID: \"7ac36a59-81e9-4274-b40d-870331ee4c34\") "
Nov 03 23:01:42.729439 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:42.729421    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ac36a59-81e9-4274-b40d-870331ee4c34-tls-certs\" (UniqueName: \"kubernetes.io/secret/7ac36a59-81e9-4274-b40d-870331ee4c34-tls-certs\") pod \"7ac36a59-81e9-4274-b40d-870331ee4c34\" (UID: \"7ac36a59-81e9-4274-b40d-870331ee4c34\") "
Nov 03 23:01:42.729693 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:42.729532    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ac36a59-81e9-4274-b40d-870331ee4c34-home\" (UniqueName: \"kubernetes.io/empty-dir/7ac36a59-81e9-4274-b40d-870331ee4c34-home\") pod \"7ac36a59-81e9-4274-b40d-870331ee4c34\" (UID: \"7ac36a59-81e9-4274-b40d-870331ee4c34\") "
Nov 03 23:01:42.730141 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:42.730111    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ac36a59-81e9-4274-b40d-870331ee4c34-home" pod "7ac36a59-81e9-4274-b40d-870331ee4c34" (UID: "7ac36a59-81e9-4274-b40d-870331ee4c34"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:01:42.730344 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:42.730286    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ac36a59-81e9-4274-b40d-870331ee4c34-tmp-dir" pod "7ac36a59-81e9-4274-b40d-870331ee4c34" (UID: "7ac36a59-81e9-4274-b40d-870331ee4c34"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:01:42.730774 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:42.730538    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ac36a59-81e9-4274-b40d-870331ee4c34-model-cache" pod "7ac36a59-81e9-4274-b40d-870331ee4c34" (UID: "7ac36a59-81e9-4274-b40d-870331ee4c34"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:01:42.744261 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:42.738952    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ac36a59-81e9-4274-b40d-870331ee4c34-tls-certs" pod "7ac36a59-81e9-4274-b40d-870331ee4c34" (UID: "7ac36a59-81e9-4274-b40d-870331ee4c34"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:01:42.744261 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:42.739003    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ac36a59-81e9-4274-b40d-870331ee4c34-kube-api-access-45jgg" pod "7ac36a59-81e9-4274-b40d-870331ee4c34" (UID: "7ac36a59-81e9-4274-b40d-870331ee4c34"). InnerVolumeSpecName "kube-api-access-45jgg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:01:42.744261 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:42.739118    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ac36a59-81e9-4274-b40d-870331ee4c34-dshm" pod "7ac36a59-81e9-4274-b40d-870331ee4c34" (UID: "7ac36a59-81e9-4274-b40d-870331ee4c34"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:01:42.830780 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:42.830704    4431 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ac36a59-81e9-4274-b40d-870331ee4c34-model-cache\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:01:42.830780 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:42.830733    4431 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ac36a59-81e9-4274-b40d-870331ee4c34-tls-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:01:42.830780 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:42.830742    4431 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ac36a59-81e9-4274-b40d-870331ee4c34-home\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:01:42.830780 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:42.830766    4431 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ac36a59-81e9-4274-b40d-870331ee4c34-tmp-dir\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:01:42.830780 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:42.830776    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45jgg\" (UniqueName: \"kubernetes.io/projected/7ac36a59-81e9-4274-b40d-870331ee4c34-kube-api-access-45jgg\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:01:42.831071 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:42.830786    4431 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ac36a59-81e9-4274-b40d-870331ee4c34-dshm\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:01:43.040496 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:43.040464    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-a940d5/sub-test-model-a940d5-kserve-656c9f7b4c-k7tz5"]
Nov 03 23:01:43.045229 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:43.045192    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-a940d5/sub-test-model-a940d5-kserve-656c9f7b4c-k7tz5"]
Nov 03 23:01:44.088744 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:44.088707    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ac36a59-81e9-4274-b40d-870331ee4c34" path="/var/lib/kubelet/pods/7ac36a59-81e9-4274-b40d-870331ee4c34/volumes"
Nov 03 23:01:44.141155 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:44.141123    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:01:45.825094 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:45.825062    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:01:47.727793 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:47.727757    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:01:49.413566 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:49.413517    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:01:50.232762 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:50.232730    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:01:51.031006 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:51.030970    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:01:52.555133 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:52.553422    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e7a7-5fc59b578-928wg"]
Nov 03 23:01:52.555133 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:52.553675    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e7a7-5fc59b578-928wg" podUID="9684a170-ee87-4c62-943f-86816632125e" containerName="maas-api" containerID="cri-o://4296695b57061563991dd274ee13b832680949eddc5a9b9569f6935b55ada396" gracePeriod=30
Nov 03 23:01:52.657794 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:52.657759    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:01:52.770677 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:52.770643    4431 generic.go:356] "Generic (PLEG): container finished" podID="9684a170-ee87-4c62-943f-86816632125e" containerID="4296695b57061563991dd274ee13b832680949eddc5a9b9569f6935b55ada396" exitCode=0
Nov 03 23:01:52.770945 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:52.770671    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e7a7-5fc59b578-928wg" event={"ID":"9684a170-ee87-4c62-943f-86816632125e","Type":"ContainerDied","Data":"4296695b57061563991dd274ee13b832680949eddc5a9b9569f6935b55ada396"}
Nov 03 23:01:52.837793 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:52.837691    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:01:52.910975 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:52.910942    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e7a7-5fc59b578-928wg"
Nov 03 23:01:52.966004 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:52.965971    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9684a170-ee87-4c62-943f-86816632125e-kube-api-access-6xgtr\" (UniqueName: \"kubernetes.io/projected/9684a170-ee87-4c62-943f-86816632125e-kube-api-access-6xgtr\") pod \"9684a170-ee87-4c62-943f-86816632125e\" (UID: \"9684a170-ee87-4c62-943f-86816632125e\") "
Nov 03 23:01:52.966004 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:52.966012    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9684a170-ee87-4c62-943f-86816632125e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9684a170-ee87-4c62-943f-86816632125e-metrics-certs\") pod \"9684a170-ee87-4c62-943f-86816632125e\" (UID: \"9684a170-ee87-4c62-943f-86816632125e\") "
Nov 03 23:01:52.966336 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:52.966134    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9684a170-ee87-4c62-943f-86816632125e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9684a170-ee87-4c62-943f-86816632125e-openshift-service-ca\") pod \"9684a170-ee87-4c62-943f-86816632125e\" (UID: \"9684a170-ee87-4c62-943f-86816632125e\") "
Nov 03 23:01:52.966336 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:52.966228    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9684a170-ee87-4c62-943f-86816632125e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9684a170-ee87-4c62-943f-86816632125e-maas-api-tls\") pod \"9684a170-ee87-4c62-943f-86816632125e\" (UID: \"9684a170-ee87-4c62-943f-86816632125e\") "
Nov 03 23:01:52.966582 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:52.966545    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9684a170-ee87-4c62-943f-86816632125e-openshift-service-ca" pod "9684a170-ee87-4c62-943f-86816632125e" (UID: "9684a170-ee87-4c62-943f-86816632125e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:01:52.966806 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:52.966789    4431 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9684a170-ee87-4c62-943f-86816632125e-openshift-service-ca\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:01:52.968846 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:52.968814    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9684a170-ee87-4c62-943f-86816632125e-metrics-certs" pod "9684a170-ee87-4c62-943f-86816632125e" (UID: "9684a170-ee87-4c62-943f-86816632125e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:01:52.969251 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:52.969190    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9684a170-ee87-4c62-943f-86816632125e-maas-api-tls" pod "9684a170-ee87-4c62-943f-86816632125e" (UID: "9684a170-ee87-4c62-943f-86816632125e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:01:52.969610 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:52.969590    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9684a170-ee87-4c62-943f-86816632125e-kube-api-access-6xgtr" pod "9684a170-ee87-4c62-943f-86816632125e" (UID: "9684a170-ee87-4c62-943f-86816632125e"). InnerVolumeSpecName "kube-api-access-6xgtr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:01:52.980174 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:52.980143    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:01:53.068770 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:53.068734    4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9684a170-ee87-4c62-943f-86816632125e-maas-api-tls\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:01:53.068770 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:53.068771    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xgtr\" (UniqueName: \"kubernetes.io/projected/9684a170-ee87-4c62-943f-86816632125e-kube-api-access-6xgtr\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:01:53.069025 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:53.068786    4431 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9684a170-ee87-4c62-943f-86816632125e-metrics-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:01:53.269964 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:53.269676    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:01:53.416922 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:53.416880    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:01:53.644298 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:53.644179    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:01:53.779506 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:53.779468    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e7a7-5fc59b578-928wg" event={"ID":"9684a170-ee87-4c62-943f-86816632125e","Type":"ContainerDied","Data":"531d3cf2b3fdef1e685afb73259462477244d1af9cd8d22428adb37c6672d069"}
Nov 03 23:01:53.779506 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:53.779495    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e7a7-5fc59b578-928wg"
Nov 03 23:01:53.779737 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:53.779539    4431 scope.go:122] "RemoveContainer" containerID="4296695b57061563991dd274ee13b832680949eddc5a9b9569f6935b55ada396"
Nov 03 23:01:53.798573 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:53.798548    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:01:53.833766 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:53.833736    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e7a7-5fc59b578-928wg"]
Nov 03 23:01:53.845499 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:53.845473    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e7a7-5fc59b578-928wg"]
Nov 03 23:01:53.954434 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:53.954405    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:01:54.089713 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:54.089679    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9684a170-ee87-4c62-943f-86816632125e" path="/var/lib/kubelet/pods/9684a170-ee87-4c62-943f-86816632125e/volumes"
Nov 03 23:01:54.407555 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:54.407469    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:01:55.131497 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:55.131454    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:01:55.292350 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:55.292312    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:01:57.222273 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:57.222196    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:01:58.898296 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:01:58.898252    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:02:00.658933 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:00.658879    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:02:00.916166 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:00.916132    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:02:01.078590 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:01.078552    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:02:01.244034 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:01.243955    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:02:01.262836 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:02:01.262801    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 23:02:01.886413 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:01.886375    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:02:02.038519 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:02.038479    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:02:02.200076 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:02.200024    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4a01f9-f7d5dd8f4-kzgmd"]
Nov 03 23:02:02.200775 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:02.200754    4431 cpu_manager.go:414] "containerMap: removing container" podUID="e02c5146-c15c-4b43-9448-457efdde2d2c" containerName="payload-pre-processing"
Nov 03 23:02:02.200775 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:02.200775    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e02c5146-c15c-4b43-9448-457efdde2d2c" containerName="payload-pre-processing"
Nov 03 23:02:02.200995 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:02.200802    4431 cpu_manager.go:414] "containerMap: removing container" podUID="9684a170-ee87-4c62-943f-86816632125e" containerName="maas-api"
Nov 03 23:02:02.200995 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:02.200810    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9684a170-ee87-4c62-943f-86816632125e" containerName="maas-api"
Nov 03 23:02:02.200995 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:02.200837    4431 cpu_manager.go:414] "containerMap: removing container" podUID="d1b37d17-4abd-408b-a1dc-b89eefeb3393" containerName="payload-processing"
Nov 03 23:02:02.200995 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:02.200844    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1b37d17-4abd-408b-a1dc-b89eefeb3393" containerName="payload-processing"
Nov 03 23:02:02.200995 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:02.200871    4431 cpu_manager.go:414] "containerMap: removing container" podUID="681f7328-e1f5-4544-876d-410f30ee83e5" containerName="istio-proxy"
Nov 03 23:02:02.200995 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:02.200878    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="681f7328-e1f5-4544-876d-410f30ee83e5" containerName="istio-proxy"
Nov 03 23:02:02.200995 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:02.200903    4431 cpu_manager.go:414] "containerMap: removing container" podUID="7ac36a59-81e9-4274-b40d-870331ee4c34" containerName="main"
Nov 03 23:02:02.200995 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:02.200910    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ac36a59-81e9-4274-b40d-870331ee4c34" containerName="main"
Nov 03 23:02:02.200995 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:02.200921    4431 cpu_manager.go:414] "containerMap: removing container" podUID="681f7328-e1f5-4544-876d-410f30ee83e5" containerName="istio-proxy"
Nov 03 23:02:02.200995 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:02.200928    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="681f7328-e1f5-4544-876d-410f30ee83e5" containerName="istio-proxy"
Nov 03 23:02:02.201591 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:02.201088    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ac36a59-81e9-4274-b40d-870331ee4c34" containerName="main"
Nov 03 23:02:02.201591 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:02.201101    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="681f7328-e1f5-4544-876d-410f30ee83e5" containerName="istio-proxy"
Nov 03 23:02:02.201591 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:02.201114    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="e02c5146-c15c-4b43-9448-457efdde2d2c" containerName="payload-pre-processing"
Nov 03 23:02:02.201591 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:02.201127    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="9684a170-ee87-4c62-943f-86816632125e" containerName="maas-api"
Nov 03 23:02:02.201591 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:02.201139    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1b37d17-4abd-408b-a1dc-b89eefeb3393" containerName="payload-processing"
Nov 03 23:02:02.206485 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:02.206458    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4a01f9-f7d5dd8f4-kzgmd"
Nov 03 23:02:02.210555 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:02.210517    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:02:02.261707 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:02.237677    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4a01f9-f7d5dd8f4-kzgmd"]
Nov 03 23:02:02.266475 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:02.266444    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-4a01f9-dockercfg-pzb9q\""
Nov 03 23:02:02.267515 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:02.267489    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-4a01f9\""
Nov 03 23:02:02.308391 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:02.308359    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j8zt\" (UniqueName: \"kubernetes.io/projected/31bea141-9c7f-44eb-98fa-86cb9d0b636a-kube-api-access-7j8zt\") pod \"payload-processing-e2e-ipp-b-4a01f9-f7d5dd8f4-kzgmd\" (UID: \"31bea141-9c7f-44eb-98fa-86cb9d0b636a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4a01f9-f7d5dd8f4-kzgmd"
Nov 03 23:02:02.308640 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:02.308617    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31bea141-9c7f-44eb-98fa-86cb9d0b636a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-4a01f9-f7d5dd8f4-kzgmd\" (UID: \"31bea141-9c7f-44eb-98fa-86cb9d0b636a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4a01f9-f7d5dd8f4-kzgmd"
Nov 03 23:02:02.409452 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:02.409411    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31bea141-9c7f-44eb-98fa-86cb9d0b636a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-4a01f9-f7d5dd8f4-kzgmd\" (UID: \"31bea141-9c7f-44eb-98fa-86cb9d0b636a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4a01f9-f7d5dd8f4-kzgmd"
Nov 03 23:02:02.409667 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:02.409519    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j8zt\" (UniqueName: \"kubernetes.io/projected/31bea141-9c7f-44eb-98fa-86cb9d0b636a-kube-api-access-7j8zt\") pod \"payload-processing-e2e-ipp-b-4a01f9-f7d5dd8f4-kzgmd\" (UID: \"31bea141-9c7f-44eb-98fa-86cb9d0b636a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4a01f9-f7d5dd8f4-kzgmd"
Nov 03 23:02:02.410202 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:02.410175    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31bea141-9c7f-44eb-98fa-86cb9d0b636a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-4a01f9-f7d5dd8f4-kzgmd\" (UID: \"31bea141-9c7f-44eb-98fa-86cb9d0b636a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4a01f9-f7d5dd8f4-kzgmd"
Nov 03 23:02:02.457405 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:02.457324    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j8zt\" (UniqueName: \"kubernetes.io/projected/31bea141-9c7f-44eb-98fa-86cb9d0b636a-kube-api-access-7j8zt\") pod \"payload-processing-e2e-ipp-b-4a01f9-f7d5dd8f4-kzgmd\" (UID: \"31bea141-9c7f-44eb-98fa-86cb9d0b636a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4a01f9-f7d5dd8f4-kzgmd"
Nov 03 23:02:02.587710 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:02.587667    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4a01f9-f7d5dd8f4-kzgmd"
Nov 03 23:02:02.895366 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:02.895337    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4a01f9-f7d5dd8f4-kzgmd"]
Nov 03 23:02:02.896973 ip-10-0-139-73 kubenswrapper[4431]: W1103 23:02:02.896944    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31bea141_9c7f_44eb_98fa_86cb9d0b636a.slice/crio-e641a1f4a210d4e4800d53e98c35ffac31414529a9576462be3062bf8b1c94f9 WatchSource:0}: Error finding container e641a1f4a210d4e4800d53e98c35ffac31414529a9576462be3062bf8b1c94f9: Status 404 returned error can't find the container with id e641a1f4a210d4e4800d53e98c35ffac31414529a9576462be3062bf8b1c94f9
Nov 03 23:02:03.013887 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:03.013854    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:02:03.850930 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:03.850889    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4a01f9-f7d5dd8f4-kzgmd" event={"ID":"31bea141-9c7f-44eb-98fa-86cb9d0b636a","Type":"ContainerStarted","Data":"e641a1f4a210d4e4800d53e98c35ffac31414529a9576462be3062bf8b1c94f9"}
Nov 03 23:02:04.515738 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:04.515702    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:02:04.861547 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:04.861452    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4a01f9-f7d5dd8f4-kzgmd" event={"ID":"31bea141-9c7f-44eb-98fa-86cb9d0b636a","Type":"ContainerStarted","Data":"1cbd6197f36f9630465422d9bd5d462f957e65190cd5e92bbf0c2d0c62c613bd"}
Nov 03 23:02:04.861711 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:04.861648    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-4a01f9-f7d5dd8f4-kzgmd"
Nov 03 23:02:04.905642 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:04.905584    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-4a01f9-f7d5dd8f4-kzgmd" podStartSLOduration=1.788116824 podStartE2EDuration="2.905570088s" totalImagesPullingTime="1.117453264s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:02:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:02:04.902750208 +0000 UTC m=+2948.928606072" watchObservedRunningTime="2026-11-03 23:02:04.905570088 +0000 UTC m=+2948.931425975"
Nov 03 23:02:06.129181 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:06.129140    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:02:07.800492 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:07.800455    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:02:09.315096 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:09.315059    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:02:10.939732 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:10.939701    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:02:12.801423 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:12.801389    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:02:12.945043 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:12.945009    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:02:14.468571 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:14.468540    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:02:15.741562 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:15.741522    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:02:15.870922 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:15.870887    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-4a01f9-f7d5dd8f4-kzgmd"
Nov 03 23:02:15.885792 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:15.885761    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:02:16.799862 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:16.799831    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:02:18.448477 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:18.448427    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:02:20.524836 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:20.524803    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:02:22.219080 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:22.219040    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:02:23.712293 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:23.712252    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-162d7b-6c77f794d7-6cqjh"]
Nov 03 23:02:23.712991 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:23.712976    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="681f7328-e1f5-4544-876d-410f30ee83e5" containerName="istio-proxy"
Nov 03 23:02:23.716871 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:23.716852    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-162d7b-6c77f794d7-6cqjh"
Nov 03 23:02:23.733416 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:23.733374    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-162d7b-serving-cert\""
Nov 03 23:02:23.743297 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:23.743258    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-162d7b-6c77f794d7-6cqjh"]
Nov 03 23:02:23.801097 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:23.801060    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d83968f-285d-4804-a015-7bddf4586765-metrics-certs\") pod \"maas-api-e2e-ipp-b-162d7b-6c77f794d7-6cqjh\" (UID: \"0d83968f-285d-4804-a015-7bddf4586765\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-162d7b-6c77f794d7-6cqjh"
Nov 03 23:02:23.801328 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:23.801132    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d83968f-285d-4804-a015-7bddf4586765-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-162d7b-6c77f794d7-6cqjh\" (UID: \"0d83968f-285d-4804-a015-7bddf4586765\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-162d7b-6c77f794d7-6cqjh"
Nov 03 23:02:23.801328 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:23.801242    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d83968f-285d-4804-a015-7bddf4586765-maas-api-tls\") pod \"maas-api-e2e-ipp-b-162d7b-6c77f794d7-6cqjh\" (UID: \"0d83968f-285d-4804-a015-7bddf4586765\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-162d7b-6c77f794d7-6cqjh"
Nov 03 23:02:23.801328 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:23.801294    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw8mn\" (UniqueName: \"kubernetes.io/projected/0d83968f-285d-4804-a015-7bddf4586765-kube-api-access-cw8mn\") pod \"maas-api-e2e-ipp-b-162d7b-6c77f794d7-6cqjh\" (UID: \"0d83968f-285d-4804-a015-7bddf4586765\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-162d7b-6c77f794d7-6cqjh"
Nov 03 23:02:23.834034 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:23.834004    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:02:23.902996 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:23.902958    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d83968f-285d-4804-a015-7bddf4586765-metrics-certs\") pod \"maas-api-e2e-ipp-b-162d7b-6c77f794d7-6cqjh\" (UID: \"0d83968f-285d-4804-a015-7bddf4586765\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-162d7b-6c77f794d7-6cqjh"
Nov 03 23:02:23.903178 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:23.903017    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d83968f-285d-4804-a015-7bddf4586765-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-162d7b-6c77f794d7-6cqjh\" (UID: \"0d83968f-285d-4804-a015-7bddf4586765\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-162d7b-6c77f794d7-6cqjh"
Nov 03 23:02:23.903178 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:23.903053    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d83968f-285d-4804-a015-7bddf4586765-maas-api-tls\") pod \"maas-api-e2e-ipp-b-162d7b-6c77f794d7-6cqjh\" (UID: \"0d83968f-285d-4804-a015-7bddf4586765\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-162d7b-6c77f794d7-6cqjh"
Nov 03 23:02:23.903178 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:23.903090    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw8mn\" (UniqueName: \"kubernetes.io/projected/0d83968f-285d-4804-a015-7bddf4586765-kube-api-access-cw8mn\") pod \"maas-api-e2e-ipp-b-162d7b-6c77f794d7-6cqjh\" (UID: \"0d83968f-285d-4804-a015-7bddf4586765\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-162d7b-6c77f794d7-6cqjh"
Nov 03 23:02:23.903822 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:23.903798    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d83968f-285d-4804-a015-7bddf4586765-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-162d7b-6c77f794d7-6cqjh\" (UID: \"0d83968f-285d-4804-a015-7bddf4586765\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-162d7b-6c77f794d7-6cqjh"
Nov 03 23:02:23.905522 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:23.905502    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d83968f-285d-4804-a015-7bddf4586765-metrics-certs\") pod \"maas-api-e2e-ipp-b-162d7b-6c77f794d7-6cqjh\" (UID: \"0d83968f-285d-4804-a015-7bddf4586765\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-162d7b-6c77f794d7-6cqjh"
Nov 03 23:02:23.905625 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:23.905540    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d83968f-285d-4804-a015-7bddf4586765-maas-api-tls\") pod \"maas-api-e2e-ipp-b-162d7b-6c77f794d7-6cqjh\" (UID: \"0d83968f-285d-4804-a015-7bddf4586765\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-162d7b-6c77f794d7-6cqjh"
Nov 03 23:02:23.927573 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:23.927536    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw8mn\" (UniqueName: \"kubernetes.io/projected/0d83968f-285d-4804-a015-7bddf4586765-kube-api-access-cw8mn\") pod \"maas-api-e2e-ipp-b-162d7b-6c77f794d7-6cqjh\" (UID: \"0d83968f-285d-4804-a015-7bddf4586765\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-162d7b-6c77f794d7-6cqjh"
Nov 03 23:02:23.976344 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:23.976244    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:02:24.031577 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:24.031545    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-162d7b-6c77f794d7-6cqjh"
Nov 03 23:02:24.341120 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:24.341100    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-162d7b-6c77f794d7-6cqjh"]
Nov 03 23:02:24.343156 ip-10-0-139-73 kubenswrapper[4431]: W1103 23:02:24.343124    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d83968f_285d_4804_a015_7bddf4586765.slice/crio-64b0e5377517596575b7507c87ae83f3a1685e4afeec63dccb495249e00cb0ed WatchSource:0}: Error finding container 64b0e5377517596575b7507c87ae83f3a1685e4afeec63dccb495249e00cb0ed: Status 404 returned error can't find the container with id 64b0e5377517596575b7507c87ae83f3a1685e4afeec63dccb495249e00cb0ed
Nov 03 23:02:24.997653 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:24.997562    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-162d7b-6c77f794d7-6cqjh" event={"ID":"0d83968f-285d-4804-a015-7bddf4586765","Type":"ContainerStarted","Data":"f8abe050bdc6c3d99b320e93888b59702ed9332bbe1432ea52f56526cb75b2a5"}
Nov 03 23:02:24.997653 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:24.997625    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-162d7b-6c77f794d7-6cqjh" event={"ID":"0d83968f-285d-4804-a015-7bddf4586765","Type":"ContainerStarted","Data":"64b0e5377517596575b7507c87ae83f3a1685e4afeec63dccb495249e00cb0ed"}
Nov 03 23:02:24.997653 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:24.997655    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-162d7b-6c77f794d7-6cqjh"
Nov 03 23:02:25.045971 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:25.045915    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-162d7b-6c77f794d7-6cqjh" podStartSLOduration=1.6794859949999998 podStartE2EDuration="2.045899975s" totalImagesPullingTime="366.41398ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:02:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:02:25.043644696 +0000 UTC m=+2969.069500559" watchObservedRunningTime="2026-11-03 23:02:25.045899975 +0000 UTC m=+2969.071755837"
Nov 03 23:02:25.599379 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:25.599347    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:02:27.329641 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:27.329602    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:02:28.847955 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:28.847915    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:02:30.351610 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:30.351568    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:02:31.015968 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:31.015938    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-162d7b-6c77f794d7-6cqjh"
Nov 03 23:02:31.259739 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:02:31.259703    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 23:02:31.814947 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:31.814911    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:02:33.414730 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:33.414689    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:02:34.876164 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:34.876130    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:02:36.044065 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:36.044037    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-e19850-788cfbf7bb-jftgq"]
Nov 03 23:02:36.044522 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:36.044225    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e19850-788cfbf7bb-jftgq" podUID="f88957ba-3271-46ca-b439-da6a010dabe9" containerName="maas-api" containerID="cri-o://4156b300e2f7ceec0c17c7b47875fe1331a7aebef00279d030b1fd7fd873f68b" gracePeriod=30
Nov 03 23:02:36.369880 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:36.369850    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e19850-788cfbf7bb-jftgq"
Nov 03 23:02:36.468207 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:36.468170    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f88957ba-3271-46ca-b439-da6a010dabe9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f88957ba-3271-46ca-b439-da6a010dabe9-openshift-service-ca\") pod \"f88957ba-3271-46ca-b439-da6a010dabe9\" (UID: \"f88957ba-3271-46ca-b439-da6a010dabe9\") "
Nov 03 23:02:36.468565 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:36.468542    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f88957ba-3271-46ca-b439-da6a010dabe9-openshift-service-ca" pod "f88957ba-3271-46ca-b439-da6a010dabe9" (UID: "f88957ba-3271-46ca-b439-da6a010dabe9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:02:36.468740 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:36.468722    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f88957ba-3271-46ca-b439-da6a010dabe9-kube-api-access-ph7nw\" (UniqueName: \"kubernetes.io/projected/f88957ba-3271-46ca-b439-da6a010dabe9-kube-api-access-ph7nw\") pod \"f88957ba-3271-46ca-b439-da6a010dabe9\" (UID: \"f88957ba-3271-46ca-b439-da6a010dabe9\") "
Nov 03 23:02:36.468829 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:36.468798    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f88957ba-3271-46ca-b439-da6a010dabe9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f88957ba-3271-46ca-b439-da6a010dabe9-maas-api-tls\") pod \"f88957ba-3271-46ca-b439-da6a010dabe9\" (UID: \"f88957ba-3271-46ca-b439-da6a010dabe9\") "
Nov 03 23:02:36.468922 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:36.468906    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f88957ba-3271-46ca-b439-da6a010dabe9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f88957ba-3271-46ca-b439-da6a010dabe9-metrics-certs\") pod \"f88957ba-3271-46ca-b439-da6a010dabe9\" (UID: \"f88957ba-3271-46ca-b439-da6a010dabe9\") "
Nov 03 23:02:36.469555 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:36.469528    4431 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f88957ba-3271-46ca-b439-da6a010dabe9-openshift-service-ca\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:02:36.478264 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:36.475857    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f88957ba-3271-46ca-b439-da6a010dabe9-maas-api-tls" pod "f88957ba-3271-46ca-b439-da6a010dabe9" (UID: "f88957ba-3271-46ca-b439-da6a010dabe9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:02:36.478264 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:36.477022    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f88957ba-3271-46ca-b439-da6a010dabe9-kube-api-access-ph7nw" pod "f88957ba-3271-46ca-b439-da6a010dabe9" (UID: "f88957ba-3271-46ca-b439-da6a010dabe9"). InnerVolumeSpecName "kube-api-access-ph7nw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:02:36.478264 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:36.477400    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f88957ba-3271-46ca-b439-da6a010dabe9-metrics-certs" pod "f88957ba-3271-46ca-b439-da6a010dabe9" (UID: "f88957ba-3271-46ca-b439-da6a010dabe9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:02:36.545433 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:36.545398    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:02:36.571023 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:36.570986    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ph7nw\" (UniqueName: \"kubernetes.io/projected/f88957ba-3271-46ca-b439-da6a010dabe9-kube-api-access-ph7nw\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:02:36.571023 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:36.571024    4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f88957ba-3271-46ca-b439-da6a010dabe9-maas-api-tls\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:02:36.571253 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:36.571037    4431 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f88957ba-3271-46ca-b439-da6a010dabe9-metrics-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:02:36.688919 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:36.688875    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:02:36.846827 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:36.846787    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:02:37.083015 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:37.082924    4431 generic.go:356] "Generic (PLEG): container finished" podID="f88957ba-3271-46ca-b439-da6a010dabe9" containerID="4156b300e2f7ceec0c17c7b47875fe1331a7aebef00279d030b1fd7fd873f68b" exitCode=0
Nov 03 23:02:37.083015 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:37.082972    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e19850-788cfbf7bb-jftgq" event={"ID":"f88957ba-3271-46ca-b439-da6a010dabe9","Type":"ContainerDied","Data":"4156b300e2f7ceec0c17c7b47875fe1331a7aebef00279d030b1fd7fd873f68b"}
Nov 03 23:02:37.083575 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:37.083025    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e19850-788cfbf7bb-jftgq" event={"ID":"f88957ba-3271-46ca-b439-da6a010dabe9","Type":"ContainerDied","Data":"f5fe50a5e01bd9236d8d550fd6399572e2c28e122a6d41fa8343341fbbe11f35"}
Nov 03 23:02:37.083575 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:37.083048    4431 scope.go:122] "RemoveContainer" containerID="4156b300e2f7ceec0c17c7b47875fe1331a7aebef00279d030b1fd7fd873f68b"
Nov 03 23:02:37.083575 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:37.082982    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e19850-788cfbf7bb-jftgq"
Nov 03 23:02:37.093483 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:37.093465    4431 scope.go:122] "RemoveContainer" containerID="4156b300e2f7ceec0c17c7b47875fe1331a7aebef00279d030b1fd7fd873f68b"
Nov 03 23:02:37.093753 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:02:37.093734    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4156b300e2f7ceec0c17c7b47875fe1331a7aebef00279d030b1fd7fd873f68b\": container with ID starting with 4156b300e2f7ceec0c17c7b47875fe1331a7aebef00279d030b1fd7fd873f68b not found: ID does not exist" containerID="4156b300e2f7ceec0c17c7b47875fe1331a7aebef00279d030b1fd7fd873f68b"
Nov 03 23:02:37.093821 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:37.093761    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4156b300e2f7ceec0c17c7b47875fe1331a7aebef00279d030b1fd7fd873f68b"} err="failed to get container status \"4156b300e2f7ceec0c17c7b47875fe1331a7aebef00279d030b1fd7fd873f68b\": rpc error: code = NotFound desc = could not find container \"4156b300e2f7ceec0c17c7b47875fe1331a7aebef00279d030b1fd7fd873f68b\": container with ID starting with 4156b300e2f7ceec0c17c7b47875fe1331a7aebef00279d030b1fd7fd873f68b not found: ID does not exist"
Nov 03 23:02:37.142342 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:37.142310    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-e19850-788cfbf7bb-jftgq"]
Nov 03 23:02:37.145269 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:37.145246    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-e19850-788cfbf7bb-jftgq"]
Nov 03 23:02:37.267375 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:37.267340    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:02:38.088077 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:38.088046    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f88957ba-3271-46ca-b439-da6a010dabe9" path="/var/lib/kubelet/pods/f88957ba-3271-46ca-b439-da6a010dabe9/volumes"
Nov 03 23:02:38.597917 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:38.597877    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:02:40.397749 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:40.397707    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:02:41.655038 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:41.655003    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4a01f9-7854647674-2mm7m"]
Nov 03 23:02:41.655574 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:41.655277    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4a01f9-7854647674-2mm7m" podUID="2b711521-cc8c-499c-90ae-6952401b15b2" containerName="maas-api" containerID="cri-o://2ac29e4812806c4e3d82af790d1a9469105f3640ab342120b4dd5ba9f6ac00ea" gracePeriod=30
Nov 03 23:02:41.748678 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:41.748648    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:02:41.807159 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:41.807124    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4a01f9-f7d5dd8f4-kzgmd"]
Nov 03 23:02:41.807677 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:41.807641    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-4a01f9-f7d5dd8f4-kzgmd" podUID="31bea141-9c7f-44eb-98fa-86cb9d0b636a" containerName="payload-processing" containerID="cri-o://1cbd6197f36f9630465422d9bd5d462f957e65190cd5e92bbf0c2d0c62c613bd" gracePeriod=30
Nov 03 23:02:41.920746 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:41.920705    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:02:42.036575 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:42.036544    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4a01f9-7854647674-2mm7m"
Nov 03 23:02:42.061835 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:42.061804    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:02:42.155243 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:42.152739    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b711521-cc8c-499c-90ae-6952401b15b2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b711521-cc8c-499c-90ae-6952401b15b2-openshift-service-ca\") pod \"2b711521-cc8c-499c-90ae-6952401b15b2\" (UID: \"2b711521-cc8c-499c-90ae-6952401b15b2\") "
Nov 03 23:02:42.155243 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:42.152927    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b711521-cc8c-499c-90ae-6952401b15b2-kube-api-access-whb28\" (UniqueName: \"kubernetes.io/projected/2b711521-cc8c-499c-90ae-6952401b15b2-kube-api-access-whb28\") pod \"2b711521-cc8c-499c-90ae-6952401b15b2\" (UID: \"2b711521-cc8c-499c-90ae-6952401b15b2\") "
Nov 03 23:02:42.155243 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:42.152965    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b711521-cc8c-499c-90ae-6952401b15b2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b711521-cc8c-499c-90ae-6952401b15b2-maas-api-tls\") pod \"2b711521-cc8c-499c-90ae-6952401b15b2\" (UID: \"2b711521-cc8c-499c-90ae-6952401b15b2\") "
Nov 03 23:02:42.155243 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:42.153065    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b711521-cc8c-499c-90ae-6952401b15b2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b711521-cc8c-499c-90ae-6952401b15b2-metrics-certs\") pod \"2b711521-cc8c-499c-90ae-6952401b15b2\" (UID: \"2b711521-cc8c-499c-90ae-6952401b15b2\") "
Nov 03 23:02:42.155243 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:42.154724    4431 generic.go:356] "Generic (PLEG): container finished" podID="31bea141-9c7f-44eb-98fa-86cb9d0b636a" containerID="1cbd6197f36f9630465422d9bd5d462f957e65190cd5e92bbf0c2d0c62c613bd" exitCode=0
Nov 03 23:02:42.155243 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:42.154820    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4a01f9-f7d5dd8f4-kzgmd" event={"ID":"31bea141-9c7f-44eb-98fa-86cb9d0b636a","Type":"ContainerDied","Data":"1cbd6197f36f9630465422d9bd5d462f957e65190cd5e92bbf0c2d0c62c613bd"}
Nov 03 23:02:42.183565 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:42.156837    4431 generic.go:356] "Generic (PLEG): container finished" podID="2b711521-cc8c-499c-90ae-6952401b15b2" containerID="2ac29e4812806c4e3d82af790d1a9469105f3640ab342120b4dd5ba9f6ac00ea" exitCode=0
Nov 03 23:02:42.183565 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:42.157058    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4a01f9-7854647674-2mm7m" event={"ID":"2b711521-cc8c-499c-90ae-6952401b15b2","Type":"ContainerDied","Data":"2ac29e4812806c4e3d82af790d1a9469105f3640ab342120b4dd5ba9f6ac00ea"}
Nov 03 23:02:42.183565 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:42.157104    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4a01f9-7854647674-2mm7m" event={"ID":"2b711521-cc8c-499c-90ae-6952401b15b2","Type":"ContainerDied","Data":"4356bed9dcc96946ca74c5639529070ba5c0a5cba9287d3da25009a85d4d9b19"}
Nov 03 23:02:42.183565 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:42.157132    4431 scope.go:122] "RemoveContainer" containerID="2ac29e4812806c4e3d82af790d1a9469105f3640ab342120b4dd5ba9f6ac00ea"
Nov 03 23:02:42.183565 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:42.157295    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4a01f9-7854647674-2mm7m"
Nov 03 23:02:42.183565 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:42.167622    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b711521-cc8c-499c-90ae-6952401b15b2-openshift-service-ca" pod "2b711521-cc8c-499c-90ae-6952401b15b2" (UID: "2b711521-cc8c-499c-90ae-6952401b15b2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:02:42.197964 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:42.196625    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b711521-cc8c-499c-90ae-6952401b15b2-metrics-certs" pod "2b711521-cc8c-499c-90ae-6952401b15b2" (UID: "2b711521-cc8c-499c-90ae-6952401b15b2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:02:42.198547 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:42.198520    4431 scope.go:122] "RemoveContainer" containerID="2ac29e4812806c4e3d82af790d1a9469105f3640ab342120b4dd5ba9f6ac00ea"
Nov 03 23:02:42.199089 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:02:42.198934    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ac29e4812806c4e3d82af790d1a9469105f3640ab342120b4dd5ba9f6ac00ea\": container with ID starting with 2ac29e4812806c4e3d82af790d1a9469105f3640ab342120b4dd5ba9f6ac00ea not found: ID does not exist" containerID="2ac29e4812806c4e3d82af790d1a9469105f3640ab342120b4dd5ba9f6ac00ea"
Nov 03 23:02:42.199089 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:42.198974    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ac29e4812806c4e3d82af790d1a9469105f3640ab342120b4dd5ba9f6ac00ea"} err="failed to get container status \"2ac29e4812806c4e3d82af790d1a9469105f3640ab342120b4dd5ba9f6ac00ea\": rpc error: code = NotFound desc = could not find container \"2ac29e4812806c4e3d82af790d1a9469105f3640ab342120b4dd5ba9f6ac00ea\": container with ID starting with 2ac29e4812806c4e3d82af790d1a9469105f3640ab342120b4dd5ba9f6ac00ea not found: ID does not exist"
Nov 03 23:02:42.215268 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:42.203711    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b711521-cc8c-499c-90ae-6952401b15b2-kube-api-access-whb28" pod "2b711521-cc8c-499c-90ae-6952401b15b2" (UID: "2b711521-cc8c-499c-90ae-6952401b15b2"). InnerVolumeSpecName "kube-api-access-whb28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:02:42.215268 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:42.211130    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b711521-cc8c-499c-90ae-6952401b15b2-maas-api-tls" pod "2b711521-cc8c-499c-90ae-6952401b15b2" (UID: "2b711521-cc8c-499c-90ae-6952401b15b2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:02:42.231063 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:42.231030    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4a01f9-f7d5dd8f4-kzgmd"
Nov 03 23:02:42.255087 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:42.255050    4431 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b711521-cc8c-499c-90ae-6952401b15b2-openshift-service-ca\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:02:42.255087 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:42.255079    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whb28\" (UniqueName: \"kubernetes.io/projected/2b711521-cc8c-499c-90ae-6952401b15b2-kube-api-access-whb28\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:02:42.255087 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:42.255092    4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b711521-cc8c-499c-90ae-6952401b15b2-maas-api-tls\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:02:42.255408 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:42.255103    4431 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b711521-cc8c-499c-90ae-6952401b15b2-metrics-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:02:42.356359 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:42.356321    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31bea141-9c7f-44eb-98fa-86cb9d0b636a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31bea141-9c7f-44eb-98fa-86cb9d0b636a-plugins-config-volume\") pod \"31bea141-9c7f-44eb-98fa-86cb9d0b636a\" (UID: \"31bea141-9c7f-44eb-98fa-86cb9d0b636a\") "
Nov 03 23:02:42.356559 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:42.356502    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31bea141-9c7f-44eb-98fa-86cb9d0b636a-kube-api-access-7j8zt\" (UniqueName: \"kubernetes.io/projected/31bea141-9c7f-44eb-98fa-86cb9d0b636a-kube-api-access-7j8zt\") pod \"31bea141-9c7f-44eb-98fa-86cb9d0b636a\" (UID: \"31bea141-9c7f-44eb-98fa-86cb9d0b636a\") "
Nov 03 23:02:42.356811 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:42.356780    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31bea141-9c7f-44eb-98fa-86cb9d0b636a-plugins-config-volume" pod "31bea141-9c7f-44eb-98fa-86cb9d0b636a" (UID: "31bea141-9c7f-44eb-98fa-86cb9d0b636a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:02:42.357203 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:42.357181    4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31bea141-9c7f-44eb-98fa-86cb9d0b636a-plugins-config-volume\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:02:42.359057 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:42.359028    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31bea141-9c7f-44eb-98fa-86cb9d0b636a-kube-api-access-7j8zt" pod "31bea141-9c7f-44eb-98fa-86cb9d0b636a" (UID: "31bea141-9c7f-44eb-98fa-86cb9d0b636a"). InnerVolumeSpecName "kube-api-access-7j8zt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:02:42.379042 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:42.378999    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:02:42.458633 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:42.458558    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7j8zt\" (UniqueName: \"kubernetes.io/projected/31bea141-9c7f-44eb-98fa-86cb9d0b636a-kube-api-access-7j8zt\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:02:42.502932 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:42.502893    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4a01f9-7854647674-2mm7m"]
Nov 03 23:02:42.506941 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:42.506910    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4a01f9-7854647674-2mm7m"]
Nov 03 23:02:42.521817 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:42.521773    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:02:42.966914 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:42.966883    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:02:43.164661 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:43.164635    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4a01f9-f7d5dd8f4-kzgmd"
Nov 03 23:02:43.164858 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:43.164672    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4a01f9-f7d5dd8f4-kzgmd" event={"ID":"31bea141-9c7f-44eb-98fa-86cb9d0b636a","Type":"ContainerDied","Data":"e641a1f4a210d4e4800d53e98c35ffac31414529a9576462be3062bf8b1c94f9"}
Nov 03 23:02:43.164858 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:43.164734    4431 scope.go:122] "RemoveContainer" containerID="1cbd6197f36f9630465422d9bd5d462f957e65190cd5e92bbf0c2d0c62c613bd"
Nov 03 23:02:43.216581 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:43.216553    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4a01f9-f7d5dd8f4-kzgmd"]
Nov 03 23:02:43.228263 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:43.228237    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4a01f9-f7d5dd8f4-kzgmd"]
Nov 03 23:02:43.836660 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:43.836627    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:02:43.981782 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:43.981711    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:02:44.096353 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:44.095258    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b711521-cc8c-499c-90ae-6952401b15b2" path="/var/lib/kubelet/pods/2b711521-cc8c-499c-90ae-6952401b15b2/volumes"
Nov 03 23:02:44.096353 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:44.095862    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31bea141-9c7f-44eb-98fa-86cb9d0b636a" path="/var/lib/kubelet/pods/31bea141-9c7f-44eb-98fa-86cb9d0b636a/volumes"
Nov 03 23:02:44.109234 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:44.105473    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96"]
Nov 03 23:02:44.109234 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:44.105699    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96" podUID="b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c" containerName="istio-proxy" containerID="cri-o://dc025a25fa12213e4c7c4dab4d84697325cbbcb38eaffcd5ce67a67128d8dae6" gracePeriod=30
Nov 03 23:02:44.170195 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:44.170140    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:02:44.321264 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:44.321226    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:02:45.109201 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:45.109166    4431 patch_prober.go:29] interesting pod/e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 23:02:45.109612 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:45.109245    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96" podUID="b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 23:02:45.809702 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:45.809664    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:02:47.492531 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:47.492493    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:02:48.940945 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:48.940910    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:02:49.081456 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:49.081420    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:02:49.262858 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:49.262771    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:02:49.471853 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:49.471827    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96"
Nov 03 23:02:49.553799 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:49.553704    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c-kube-api-access-fwhgp\" (UniqueName: \"kubernetes.io/projected/b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c-kube-api-access-fwhgp\") pod \"b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c\" (UID: \"b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c\") "
Nov 03 23:02:49.553799 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:49.553752    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c-istio-data\") pod \"b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c\" (UID: \"b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c\") "
Nov 03 23:02:49.553799 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:49.553786    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c-istio-podinfo\") pod \"b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c\" (UID: \"b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c\") "
Nov 03 23:02:49.554112 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:49.553878    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c-istio-envoy\") pod \"b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c\" (UID: \"b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c\") "
Nov 03 23:02:49.554112 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:49.553918    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c-workload-socket\") pod \"b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c\" (UID: \"b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c\") "
Nov 03 23:02:49.554112 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:49.554038    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c-workload-certs\") pod \"b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c\" (UID: \"b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c\") "
Nov 03 23:02:49.554334 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:49.554133    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c-credential-socket\") pod \"b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c\" (UID: \"b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c\") "
Nov 03 23:02:49.554334 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:49.554157    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c-istio-data" pod "b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c" (UID: "b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:02:49.554334 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:49.554295    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c-istio-token\" (UniqueName: \"kubernetes.io/projected/b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c-istio-token\") pod \"b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c\" (UID: \"b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c\") "
Nov 03 23:02:49.554334 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:49.554330    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c-istiod-ca-cert\") pod \"b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c\" (UID: \"b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c\") "
Nov 03 23:02:49.554717 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:49.554687    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c-workload-socket" pod "b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c" (UID: "b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:02:49.554819 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:49.554742    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c-workload-certs" pod "b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c" (UID: "b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:02:49.554884 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:49.554812    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c-credential-socket" pod "b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c" (UID: "b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:02:49.555063 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:49.555036    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c-istiod-ca-cert" pod "b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c" (UID: "b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:02:49.555190 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:49.555096    4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c-workload-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:02:49.555190 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:49.555121    4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c-workload-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:02:49.555190 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:49.555135    4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c-credential-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:02:49.555190 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:49.555150    4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c-istio-data\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:02:49.556326 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:49.556301    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c-kube-api-access-fwhgp" pod "b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c" (UID: "b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c"). InnerVolumeSpecName "kube-api-access-fwhgp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:02:49.556662 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:49.556645    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c-istio-podinfo" pod "b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c" (UID: "b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 23:02:49.556976 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:49.556955    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c-istio-token" pod "b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c" (UID: "b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:02:49.557085 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:49.557065    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c-istio-envoy" pod "b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c" (UID: "b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:02:49.656130 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:49.656101    4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c-istio-token\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:02:49.656130 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:49.656128    4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c-istiod-ca-cert\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:02:49.656130 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:49.656137    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwhgp\" (UniqueName: \"kubernetes.io/projected/b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c-kube-api-access-fwhgp\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:02:49.656431 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:49.656148    4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c-istio-podinfo\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:02:49.656431 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:49.656157    4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c-istio-envoy\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:02:50.215787 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:50.215752    4431 generic.go:356] "Generic (PLEG): container finished" podID="b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c" containerID="dc025a25fa12213e4c7c4dab4d84697325cbbcb38eaffcd5ce67a67128d8dae6" exitCode=0
Nov 03 23:02:50.216285 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:50.215806    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96"
Nov 03 23:02:50.216285 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:50.215838    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96" event={"ID":"b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c","Type":"ContainerDied","Data":"dc025a25fa12213e4c7c4dab4d84697325cbbcb38eaffcd5ce67a67128d8dae6"}
Nov 03 23:02:50.216285 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:50.215906    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96" event={"ID":"b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c","Type":"ContainerDied","Data":"98689f08b499e711c88d128c6b0592992e3236068aee01afde16d27d51caff34"}
Nov 03 23:02:50.216285 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:50.215940    4431 scope.go:122] "RemoveContainer" containerID="dc025a25fa12213e4c7c4dab4d84697325cbbcb38eaffcd5ce67a67128d8dae6"
Nov 03 23:02:50.225910 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:50.225878    4431 scope.go:122] "RemoveContainer" containerID="be9f70a522fa1e9bb79e5fde7dfc3cce48574a605cdbf375d5d5807d2e025250"
Nov 03 23:02:50.235307 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:50.235290    4431 scope.go:122] "RemoveContainer" containerID="dc025a25fa12213e4c7c4dab4d84697325cbbcb38eaffcd5ce67a67128d8dae6"
Nov 03 23:02:50.235599 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:02:50.235573    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc025a25fa12213e4c7c4dab4d84697325cbbcb38eaffcd5ce67a67128d8dae6\": container with ID starting with dc025a25fa12213e4c7c4dab4d84697325cbbcb38eaffcd5ce67a67128d8dae6 not found: ID does not exist" containerID="dc025a25fa12213e4c7c4dab4d84697325cbbcb38eaffcd5ce67a67128d8dae6"
Nov 03 23:02:50.235711 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:50.235606    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc025a25fa12213e4c7c4dab4d84697325cbbcb38eaffcd5ce67a67128d8dae6"} err="failed to get container status \"dc025a25fa12213e4c7c4dab4d84697325cbbcb38eaffcd5ce67a67128d8dae6\": rpc error: code = NotFound desc = could not find container \"dc025a25fa12213e4c7c4dab4d84697325cbbcb38eaffcd5ce67a67128d8dae6\": container with ID starting with dc025a25fa12213e4c7c4dab4d84697325cbbcb38eaffcd5ce67a67128d8dae6 not found: ID does not exist"
Nov 03 23:02:50.235711 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:50.235626    4431 scope.go:122] "RemoveContainer" containerID="be9f70a522fa1e9bb79e5fde7dfc3cce48574a605cdbf375d5d5807d2e025250"
Nov 03 23:02:50.235927 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:02:50.235907    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be9f70a522fa1e9bb79e5fde7dfc3cce48574a605cdbf375d5d5807d2e025250\": container with ID starting with be9f70a522fa1e9bb79e5fde7dfc3cce48574a605cdbf375d5d5807d2e025250 not found: ID does not exist" containerID="be9f70a522fa1e9bb79e5fde7dfc3cce48574a605cdbf375d5d5807d2e025250"
Nov 03 23:02:50.235974 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:50.235932    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be9f70a522fa1e9bb79e5fde7dfc3cce48574a605cdbf375d5d5807d2e025250"} err="failed to get container status \"be9f70a522fa1e9bb79e5fde7dfc3cce48574a605cdbf375d5d5807d2e025250\": rpc error: code = NotFound desc = could not find container \"be9f70a522fa1e9bb79e5fde7dfc3cce48574a605cdbf375d5d5807d2e025250\": container with ID starting with be9f70a522fa1e9bb79e5fde7dfc3cce48574a605cdbf375d5d5807d2e025250 not found: ID does not exist"
Nov 03 23:02:50.279643 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:50.279614    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96"]
Nov 03 23:02:50.291453 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:50.291425    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-4kq96"]
Nov 03 23:02:50.374431 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:50.374334    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:02:50.668933 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:50.668897    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:02:52.088559 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:52.088525    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c" path="/var/lib/kubelet/pods/b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c/volumes"
Nov 03 23:02:52.341009 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:52.340930    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:02:53.868084 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:53.868047    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:02:55.430517 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:55.430479    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:02:56.122431 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:56.122391    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:02:56.919889 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:56.919858    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mx5q8_ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea/ovn-acl-logging/0.log"
Nov 03 23:02:56.928632 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:56.928606    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mx5q8_ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea/ovn-acl-logging/0.log"
Nov 03 23:02:57.730266 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:57.730229    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:02:59.041211 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:59.041178    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:02:59.182630 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:59.182593    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:02:59.326490 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:59.326404    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:02:59.791033 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:02:59.790997    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:03:00.353383 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:00.353349    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-d4b6bb-75d4df64b8-nvlff"]
Nov 03 23:03:00.627948 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:00.627845    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:03:00.773580 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:00.773543    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:03:01.260140 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:03:01.260100    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 23:03:01.521753 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:01.521665    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:03:03.307780 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:03.307747    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:03:05.160152 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:05.160110    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:03:05.353975 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:05.353933    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d4b6bb-75d4df64b8-nvlff" podUID="60d398e2-9692-4010-83ed-d660486bc982" containerName="payload-processing" containerID="cri-o://5434b5113a9b565afc150107bdccb9debcebfdeb0e7f548cb2348a82dd4929c8" gracePeriod=25
Nov 03 23:03:05.619513 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:05.619483    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d4b6bb-75d4df64b8-nvlff"
Nov 03 23:03:05.757722 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:05.757637    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60d398e2-9692-4010-83ed-d660486bc982-kube-api-access-b9bd9\" (UniqueName: \"kubernetes.io/projected/60d398e2-9692-4010-83ed-d660486bc982-kube-api-access-b9bd9\") pod \"60d398e2-9692-4010-83ed-d660486bc982\" (UID: \"60d398e2-9692-4010-83ed-d660486bc982\") "
Nov 03 23:03:05.757874 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:05.757780    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60d398e2-9692-4010-83ed-d660486bc982-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60d398e2-9692-4010-83ed-d660486bc982-plugins-config-volume\") pod \"60d398e2-9692-4010-83ed-d660486bc982\" (UID: \"60d398e2-9692-4010-83ed-d660486bc982\") "
Nov 03 23:03:05.758172 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:05.758144    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60d398e2-9692-4010-83ed-d660486bc982-plugins-config-volume" pod "60d398e2-9692-4010-83ed-d660486bc982" (UID: "60d398e2-9692-4010-83ed-d660486bc982"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:03:05.758507 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:05.758439    4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60d398e2-9692-4010-83ed-d660486bc982-plugins-config-volume\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:03:05.762783 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:05.762751    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60d398e2-9692-4010-83ed-d660486bc982-kube-api-access-b9bd9" pod "60d398e2-9692-4010-83ed-d660486bc982" (UID: "60d398e2-9692-4010-83ed-d660486bc982"). InnerVolumeSpecName "kube-api-access-b9bd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:03:05.859750 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:05.859717    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9bd9\" (UniqueName: \"kubernetes.io/projected/60d398e2-9692-4010-83ed-d660486bc982-kube-api-access-b9bd9\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:03:06.319964 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:06.319929    4431 generic.go:356] "Generic (PLEG): container finished" podID="60d398e2-9692-4010-83ed-d660486bc982" containerID="5434b5113a9b565afc150107bdccb9debcebfdeb0e7f548cb2348a82dd4929c8" exitCode=0
Nov 03 23:03:06.320505 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:06.320022    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d4b6bb-75d4df64b8-nvlff"
Nov 03 23:03:06.320505 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:06.320025    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d4b6bb-75d4df64b8-nvlff" event={"ID":"60d398e2-9692-4010-83ed-d660486bc982","Type":"ContainerDied","Data":"5434b5113a9b565afc150107bdccb9debcebfdeb0e7f548cb2348a82dd4929c8"}
Nov 03 23:03:06.320505 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:06.320096    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d4b6bb-75d4df64b8-nvlff" event={"ID":"60d398e2-9692-4010-83ed-d660486bc982","Type":"ContainerDied","Data":"9e0ad0f542941a5c51461ef61e3674f44b7cccac80b4f46ff98d889cbee7bc1b"}
Nov 03 23:03:06.320505 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:06.320132    4431 scope.go:122] "RemoveContainer" containerID="5434b5113a9b565afc150107bdccb9debcebfdeb0e7f548cb2348a82dd4929c8"
Nov 03 23:03:06.337938 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:06.337906    4431 scope.go:122] "RemoveContainer" containerID="5434b5113a9b565afc150107bdccb9debcebfdeb0e7f548cb2348a82dd4929c8"
Nov 03 23:03:06.338370 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:03:06.338347    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5434b5113a9b565afc150107bdccb9debcebfdeb0e7f548cb2348a82dd4929c8\": container with ID starting with 5434b5113a9b565afc150107bdccb9debcebfdeb0e7f548cb2348a82dd4929c8 not found: ID does not exist" containerID="5434b5113a9b565afc150107bdccb9debcebfdeb0e7f548cb2348a82dd4929c8"
Nov 03 23:03:06.338491 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:06.338376    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5434b5113a9b565afc150107bdccb9debcebfdeb0e7f548cb2348a82dd4929c8"} err="failed to get container status \"5434b5113a9b565afc150107bdccb9debcebfdeb0e7f548cb2348a82dd4929c8\": rpc error: code = NotFound desc = could not find container \"5434b5113a9b565afc150107bdccb9debcebfdeb0e7f548cb2348a82dd4929c8\": container with ID starting with 5434b5113a9b565afc150107bdccb9debcebfdeb0e7f548cb2348a82dd4929c8 not found: ID does not exist"
Nov 03 23:03:06.372407 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:06.372334    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-d4b6bb-75d4df64b8-nvlff"]
Nov 03 23:03:06.383797 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:06.383772    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-d4b6bb-75d4df64b8-nvlff"]
Nov 03 23:03:06.876121 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:06.876090    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:03:07.768636 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:07.768605    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:03:07.970764 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:07.970732    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:03:08.088677 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:08.088600    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60d398e2-9692-4010-83ed-d660486bc982" path="/var/lib/kubelet/pods/60d398e2-9692-4010-83ed-d660486bc982/volumes"
Nov 03 23:03:08.114992 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:08.114966    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:03:08.624491 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:08.624453    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:03:10.130880 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:10.130769    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:03:11.794912 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:11.794850    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:03:13.304444 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:13.304408    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:03:14.977414 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:14.977374    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:03:16.387944 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:16.387909    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:03:16.706380 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:16.706341    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:03:18.101862 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:18.101831    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:03:18.281813 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:18.281780    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:03:18.422320 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:18.422283    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:03:18.673515 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:18.673433    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:03:18.818854 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:18.818810    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:03:18.968406 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:18.968319    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:03:19.274855 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:19.274785    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:03:20.497151 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:20.497111    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:03:22.057541 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:22.057497    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:03:23.614972 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:23.614937    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:03:24.205306 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:24.205272    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-3920cb/ipp-route-3920cb-kserve-76df875dbf-rrqjq"]
Nov 03 23:03:24.309305 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:24.309257    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:03:25.090120 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:25.090088    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:03:26.631688 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:26.631655    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:03:28.221299 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:28.221243    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:03:29.802008 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:29.801943    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:03:31.259858 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:03:31.259815    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 23:03:31.341907 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:31.341866    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:03:32.466448 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:32.466416    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:03:32.606925 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:32.606889    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:03:32.819957 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:32.819877    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4ad693-5685dfd7bc-vbh2d"]
Nov 03 23:03:32.820484 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:32.820469    4431 cpu_manager.go:414] "containerMap: removing container" podUID="2b711521-cc8c-499c-90ae-6952401b15b2" containerName="maas-api"
Nov 03 23:03:32.820547 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:32.820485    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b711521-cc8c-499c-90ae-6952401b15b2" containerName="maas-api"
Nov 03 23:03:32.820547 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:32.820497    4431 cpu_manager.go:414] "containerMap: removing container" podUID="60d398e2-9692-4010-83ed-d660486bc982" containerName="payload-processing"
Nov 03 23:03:32.820547 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:32.820502    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60d398e2-9692-4010-83ed-d660486bc982" containerName="payload-processing"
Nov 03 23:03:32.820547 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:32.820520    4431 cpu_manager.go:414] "containerMap: removing container" podUID="b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c" containerName="istio-proxy"
Nov 03 23:03:32.820547 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:32.820525    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c" containerName="istio-proxy"
Nov 03 23:03:32.820547 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:32.820535    4431 cpu_manager.go:414] "containerMap: removing container" podUID="31bea141-9c7f-44eb-98fa-86cb9d0b636a" containerName="payload-processing"
Nov 03 23:03:32.820547 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:32.820540    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31bea141-9c7f-44eb-98fa-86cb9d0b636a" containerName="payload-processing"
Nov 03 23:03:32.820784 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:32.820569    4431 cpu_manager.go:414] "containerMap: removing container" podUID="f88957ba-3271-46ca-b439-da6a010dabe9" containerName="maas-api"
Nov 03 23:03:32.820784 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:32.820573    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f88957ba-3271-46ca-b439-da6a010dabe9" containerName="maas-api"
Nov 03 23:03:32.820784 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:32.820656    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c" containerName="istio-proxy"
Nov 03 23:03:32.820784 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:32.820664    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="f88957ba-3271-46ca-b439-da6a010dabe9" containerName="maas-api"
Nov 03 23:03:32.820784 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:32.820671    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b711521-cc8c-499c-90ae-6952401b15b2" containerName="maas-api"
Nov 03 23:03:32.820784 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:32.820678    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="60d398e2-9692-4010-83ed-d660486bc982" containerName="payload-processing"
Nov 03 23:03:32.820784 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:32.820687    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="31bea141-9c7f-44eb-98fa-86cb9d0b636a" containerName="payload-processing"
Nov 03 23:03:32.824318 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:32.824299    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ad693-5685dfd7bc-vbh2d"
Nov 03 23:03:32.843691 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:32.843665    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4ad693-5685dfd7bc-vbh2d"]
Nov 03 23:03:32.851103 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:32.851079    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-4ad693-dockercfg-fr598\""
Nov 03 23:03:32.851103 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:32.851108    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-4ad693\""
Nov 03 23:03:32.904341 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:32.904308    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afd0d809-1add-4229-aa4f-eb5b688c2a10-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4ad693-5685dfd7bc-vbh2d\" (UID: \"afd0d809-1add-4229-aa4f-eb5b688c2a10\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ad693-5685dfd7bc-vbh2d"
Nov 03 23:03:32.904524 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:32.904502    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzfvh\" (UniqueName: \"kubernetes.io/projected/afd0d809-1add-4229-aa4f-eb5b688c2a10-kube-api-access-bzfvh\") pod \"payload-pre-processing-e2e-ipp-a-4ad693-5685dfd7bc-vbh2d\" (UID: \"afd0d809-1add-4229-aa4f-eb5b688c2a10\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ad693-5685dfd7bc-vbh2d"
Nov 03 23:03:33.005394 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:33.005358    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzfvh\" (UniqueName: \"kubernetes.io/projected/afd0d809-1add-4229-aa4f-eb5b688c2a10-kube-api-access-bzfvh\") pod \"payload-pre-processing-e2e-ipp-a-4ad693-5685dfd7bc-vbh2d\" (UID: \"afd0d809-1add-4229-aa4f-eb5b688c2a10\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ad693-5685dfd7bc-vbh2d"
Nov 03 23:03:33.005571 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:33.005375    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4ad693-589cf5d8f6-lgfqh"]
Nov 03 23:03:33.005571 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:33.005527    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afd0d809-1add-4229-aa4f-eb5b688c2a10-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4ad693-5685dfd7bc-vbh2d\" (UID: \"afd0d809-1add-4229-aa4f-eb5b688c2a10\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ad693-5685dfd7bc-vbh2d"
Nov 03 23:03:33.006493 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:33.006465    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afd0d809-1add-4229-aa4f-eb5b688c2a10-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4ad693-5685dfd7bc-vbh2d\" (UID: \"afd0d809-1add-4229-aa4f-eb5b688c2a10\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ad693-5685dfd7bc-vbh2d"
Nov 03 23:03:33.006635 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:33.006506    4431 cpu_manager.go:414] "containerMap: removing container" podUID="b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c" containerName="istio-proxy"
Nov 03 23:03:33.006635 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:33.006522    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c" containerName="istio-proxy"
Nov 03 23:03:33.006714 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:33.006675    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="b960a1f1-6b5e-4599-8e2e-3d4a13dc8b6c" containerName="istio-proxy"
Nov 03 23:03:33.010165 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:33.010149    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ad693-589cf5d8f6-lgfqh"
Nov 03 23:03:33.027634 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:33.027611    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4ad693-589cf5d8f6-lgfqh"]
Nov 03 23:03:33.033498 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:33.033474    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzfvh\" (UniqueName: \"kubernetes.io/projected/afd0d809-1add-4229-aa4f-eb5b688c2a10-kube-api-access-bzfvh\") pod \"payload-pre-processing-e2e-ipp-a-4ad693-5685dfd7bc-vbh2d\" (UID: \"afd0d809-1add-4229-aa4f-eb5b688c2a10\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ad693-5685dfd7bc-vbh2d"
Nov 03 23:03:33.107159 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:33.107067    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swps6\" (UniqueName: \"kubernetes.io/projected/da056270-6c9e-4786-8b6e-e706e6fd3dcd-kube-api-access-swps6\") pod \"payload-processing-e2e-ipp-a-4ad693-589cf5d8f6-lgfqh\" (UID: \"da056270-6c9e-4786-8b6e-e706e6fd3dcd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4ad693-589cf5d8f6-lgfqh"
Nov 03 23:03:33.107351 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:33.107297    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da056270-6c9e-4786-8b6e-e706e6fd3dcd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4ad693-589cf5d8f6-lgfqh\" (UID: \"da056270-6c9e-4786-8b6e-e706e6fd3dcd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4ad693-589cf5d8f6-lgfqh"
Nov 03 23:03:33.138593 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:33.138557    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ad693-5685dfd7bc-vbh2d"
Nov 03 23:03:33.214255 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:33.211559    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swps6\" (UniqueName: \"kubernetes.io/projected/da056270-6c9e-4786-8b6e-e706e6fd3dcd-kube-api-access-swps6\") pod \"payload-processing-e2e-ipp-a-4ad693-589cf5d8f6-lgfqh\" (UID: \"da056270-6c9e-4786-8b6e-e706e6fd3dcd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4ad693-589cf5d8f6-lgfqh"
Nov 03 23:03:33.214255 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:33.211854    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da056270-6c9e-4786-8b6e-e706e6fd3dcd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4ad693-589cf5d8f6-lgfqh\" (UID: \"da056270-6c9e-4786-8b6e-e706e6fd3dcd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4ad693-589cf5d8f6-lgfqh"
Nov 03 23:03:33.214255 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:33.214192    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da056270-6c9e-4786-8b6e-e706e6fd3dcd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4ad693-589cf5d8f6-lgfqh\" (UID: \"da056270-6c9e-4786-8b6e-e706e6fd3dcd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4ad693-589cf5d8f6-lgfqh"
Nov 03 23:03:33.236492 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:33.236455    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swps6\" (UniqueName: \"kubernetes.io/projected/da056270-6c9e-4786-8b6e-e706e6fd3dcd-kube-api-access-swps6\") pod \"payload-processing-e2e-ipp-a-4ad693-589cf5d8f6-lgfqh\" (UID: \"da056270-6c9e-4786-8b6e-e706e6fd3dcd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4ad693-589cf5d8f6-lgfqh"
Nov 03 23:03:33.324058 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:33.324024    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ad693-589cf5d8f6-lgfqh"
Nov 03 23:03:33.442938 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:33.442906    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4ad693-5685dfd7bc-vbh2d"]
Nov 03 23:03:33.444541 ip-10-0-139-73 kubenswrapper[4431]: W1103 23:03:33.444514    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafd0d809_1add_4229_aa4f_eb5b688c2a10.slice/crio-dbb4a93504c21971580a9ae50dd59b77de143f186cad32e69fb6d88fca188555 WatchSource:0}: Error finding container dbb4a93504c21971580a9ae50dd59b77de143f186cad32e69fb6d88fca188555: Status 404 returned error can't find the container with id dbb4a93504c21971580a9ae50dd59b77de143f186cad32e69fb6d88fca188555
Nov 03 23:03:33.493863 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:33.493836    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ad693-5685dfd7bc-vbh2d" event={"ID":"afd0d809-1add-4229-aa4f-eb5b688c2a10","Type":"ContainerStarted","Data":"dbb4a93504c21971580a9ae50dd59b77de143f186cad32e69fb6d88fca188555"}
Nov 03 23:03:33.550633 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:33.550596    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:03:33.626756 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:33.626725    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4ad693-589cf5d8f6-lgfqh"]
Nov 03 23:03:33.629000 ip-10-0-139-73 kubenswrapper[4431]: W1103 23:03:33.628966    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda056270_6c9e_4786_8b6e_e706e6fd3dcd.slice/crio-ab125054289681e90f0c1b40a7ec50739b4c6644ef2756570f9efaf8464ddcec WatchSource:0}: Error finding container ab125054289681e90f0c1b40a7ec50739b4c6644ef2756570f9efaf8464ddcec: Status 404 returned error can't find the container with id ab125054289681e90f0c1b40a7ec50739b4c6644ef2756570f9efaf8464ddcec
Nov 03 23:03:34.502613 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:34.502570    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ad693-5685dfd7bc-vbh2d" event={"ID":"afd0d809-1add-4229-aa4f-eb5b688c2a10","Type":"ContainerStarted","Data":"04570e9b3f067446b147eb5613f1f8d3d466c4d2d07cdbe7cff8e4d315598d24"}
Nov 03 23:03:34.503117 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:34.502639    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ad693-5685dfd7bc-vbh2d"
Nov 03 23:03:34.504459 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:34.504429    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ad693-589cf5d8f6-lgfqh" event={"ID":"da056270-6c9e-4786-8b6e-e706e6fd3dcd","Type":"ContainerStarted","Data":"db0386b4c4982e1bad405e9ca9598e262ac0b6907e690ba945d7ff58a31eb4fb"}
Nov 03 23:03:34.504581 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:34.504486    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ad693-589cf5d8f6-lgfqh"
Nov 03 23:03:34.504581 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:34.504511    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ad693-589cf5d8f6-lgfqh" event={"ID":"da056270-6c9e-4786-8b6e-e706e6fd3dcd","Type":"ContainerStarted","Data":"ab125054289681e90f0c1b40a7ec50739b4c6644ef2756570f9efaf8464ddcec"}
Nov 03 23:03:34.546254 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:34.546183    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ad693-5685dfd7bc-vbh2d" podStartSLOduration=2.546167565 podStartE2EDuration="2.546167565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:03:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:03:34.54479664 +0000 UTC m=+3038.570652507" watchObservedRunningTime="2026-11-03 23:03:34.546167565 +0000 UTC m=+3038.572023428"
Nov 03 23:03:34.584662 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:34.584612    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ad693-589cf5d8f6-lgfqh" podStartSLOduration=2.584604691 podStartE2EDuration="2.584604691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:03:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:03:34.58455787 +0000 UTC m=+3038.610413735" watchObservedRunningTime="2026-11-03 23:03:34.584604691 +0000 UTC m=+3038.610460586"
Nov 03 23:03:35.608934 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:35.608898    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:03:37.191689 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:37.191655    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:03:38.867008 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:38.866977    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:03:38.885706 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:38.885676    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d4b6bb-878757bb6-fxn94"]
Nov 03 23:03:38.885874 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:38.885854    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d4b6bb-878757bb6-fxn94" podUID="bb8f978d-2eb9-4cf4-a7ea-ca3caf0c11ab" containerName="maas-api" containerID="cri-o://797d8772836441cbbdc4f350a6e3f5d26e2a4e7f1277ad18460d955ff9d9248b" gracePeriod=30
Nov 03 23:03:39.016835 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:39.016788    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:03:39.223159 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:39.223134    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d4b6bb-878757bb6-fxn94"
Nov 03 23:03:39.226323 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:39.226293    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-3920cb/ipp-route-3920cb-kserve-76df875dbf-rrqjq" podUID="0fc68969-ebce-4cfb-bdb0-8600510e931f" containerName="main" containerID="cri-o://4585bf084388c39eeaa35cf1e4f2a9043e63ba6917678722fb9841ed01fee112" gracePeriod=45
Nov 03 23:03:39.298050 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:39.298016    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb8f978d-2eb9-4cf4-a7ea-ca3caf0c11ab-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb8f978d-2eb9-4cf4-a7ea-ca3caf0c11ab-maas-api-tls\") pod \"bb8f978d-2eb9-4cf4-a7ea-ca3caf0c11ab\" (UID: \"bb8f978d-2eb9-4cf4-a7ea-ca3caf0c11ab\") "
Nov 03 23:03:39.298255 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:39.298102    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb8f978d-2eb9-4cf4-a7ea-ca3caf0c11ab-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb8f978d-2eb9-4cf4-a7ea-ca3caf0c11ab-openshift-service-ca\") pod \"bb8f978d-2eb9-4cf4-a7ea-ca3caf0c11ab\" (UID: \"bb8f978d-2eb9-4cf4-a7ea-ca3caf0c11ab\") "
Nov 03 23:03:39.298255 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:39.298246    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb8f978d-2eb9-4cf4-a7ea-ca3caf0c11ab-kube-api-access-654tf\" (UniqueName: \"kubernetes.io/projected/bb8f978d-2eb9-4cf4-a7ea-ca3caf0c11ab-kube-api-access-654tf\") pod \"bb8f978d-2eb9-4cf4-a7ea-ca3caf0c11ab\" (UID: \"bb8f978d-2eb9-4cf4-a7ea-ca3caf0c11ab\") "
Nov 03 23:03:39.298456 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:39.298437    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb8f978d-2eb9-4cf4-a7ea-ca3caf0c11ab-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb8f978d-2eb9-4cf4-a7ea-ca3caf0c11ab-metrics-certs\") pod \"bb8f978d-2eb9-4cf4-a7ea-ca3caf0c11ab\" (UID: \"bb8f978d-2eb9-4cf4-a7ea-ca3caf0c11ab\") "
Nov 03 23:03:39.298592 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:39.298562    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb8f978d-2eb9-4cf4-a7ea-ca3caf0c11ab-openshift-service-ca" pod "bb8f978d-2eb9-4cf4-a7ea-ca3caf0c11ab" (UID: "bb8f978d-2eb9-4cf4-a7ea-ca3caf0c11ab"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:03:39.299059 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:39.299025    4431 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb8f978d-2eb9-4cf4-a7ea-ca3caf0c11ab-openshift-service-ca\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:03:39.300714 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:39.300688    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb8f978d-2eb9-4cf4-a7ea-ca3caf0c11ab-kube-api-access-654tf" pod "bb8f978d-2eb9-4cf4-a7ea-ca3caf0c11ab" (UID: "bb8f978d-2eb9-4cf4-a7ea-ca3caf0c11ab"). InnerVolumeSpecName "kube-api-access-654tf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:03:39.300876 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:39.300828    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb8f978d-2eb9-4cf4-a7ea-ca3caf0c11ab-maas-api-tls" pod "bb8f978d-2eb9-4cf4-a7ea-ca3caf0c11ab" (UID: "bb8f978d-2eb9-4cf4-a7ea-ca3caf0c11ab"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:03:39.301046 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:39.301018    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb8f978d-2eb9-4cf4-a7ea-ca3caf0c11ab-metrics-certs" pod "bb8f978d-2eb9-4cf4-a7ea-ca3caf0c11ab" (UID: "bb8f978d-2eb9-4cf4-a7ea-ca3caf0c11ab"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:03:39.400725 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:39.400282    4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb8f978d-2eb9-4cf4-a7ea-ca3caf0c11ab-maas-api-tls\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:03:39.400725 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:39.400325    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-654tf\" (UniqueName: \"kubernetes.io/projected/bb8f978d-2eb9-4cf4-a7ea-ca3caf0c11ab-kube-api-access-654tf\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:03:39.400725 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:39.400340    4431 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb8f978d-2eb9-4cf4-a7ea-ca3caf0c11ab-metrics-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:03:39.443191 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:39.443161    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:03:39.521677 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:39.521651    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-3920cb/ipp-route-3920cb-kserve-76df875dbf-rrqjq"
Nov 03 23:03:39.535345 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:39.535254    4431 generic.go:356] "Generic (PLEG): container finished" podID="bb8f978d-2eb9-4cf4-a7ea-ca3caf0c11ab" containerID="797d8772836441cbbdc4f350a6e3f5d26e2a4e7f1277ad18460d955ff9d9248b" exitCode=0
Nov 03 23:03:39.535497 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:39.535352    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d4b6bb-878757bb6-fxn94"
Nov 03 23:03:39.536020 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:39.535994    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d4b6bb-878757bb6-fxn94" event={"ID":"bb8f978d-2eb9-4cf4-a7ea-ca3caf0c11ab","Type":"ContainerDied","Data":"797d8772836441cbbdc4f350a6e3f5d26e2a4e7f1277ad18460d955ff9d9248b"}
Nov 03 23:03:39.536068 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:39.536046    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d4b6bb-878757bb6-fxn94" event={"ID":"bb8f978d-2eb9-4cf4-a7ea-ca3caf0c11ab","Type":"ContainerDied","Data":"7ffc1e6d64e946a024886cb7dad6d726722e9fc813ac5d9e7d4cd0af5d634330"}
Nov 03 23:03:39.536107 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:39.536067    4431 scope.go:122] "RemoveContainer" containerID="797d8772836441cbbdc4f350a6e3f5d26e2a4e7f1277ad18460d955ff9d9248b"
Nov 03 23:03:39.540011 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:39.539956    4431 generic.go:356] "Generic (PLEG): container finished" podID="0fc68969-ebce-4cfb-bdb0-8600510e931f" containerID="4585bf084388c39eeaa35cf1e4f2a9043e63ba6917678722fb9841ed01fee112" exitCode=0
Nov 03 23:03:39.540011 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:39.539992    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-3920cb/ipp-route-3920cb-kserve-76df875dbf-rrqjq" event={"ID":"0fc68969-ebce-4cfb-bdb0-8600510e931f","Type":"ContainerDied","Data":"4585bf084388c39eeaa35cf1e4f2a9043e63ba6917678722fb9841ed01fee112"}
Nov 03 23:03:39.540203 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:39.540040    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-3920cb/ipp-route-3920cb-kserve-76df875dbf-rrqjq" event={"ID":"0fc68969-ebce-4cfb-bdb0-8600510e931f","Type":"ContainerDied","Data":"32e68125fda8101c5b8c86c6c326526a4fb5745234d048543ddbe1b6d1f281ce"}
Nov 03 23:03:39.540203 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:39.540004    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-3920cb/ipp-route-3920cb-kserve-76df875dbf-rrqjq"
Nov 03 23:03:39.547974 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:39.547955    4431 scope.go:122] "RemoveContainer" containerID="797d8772836441cbbdc4f350a6e3f5d26e2a4e7f1277ad18460d955ff9d9248b"
Nov 03 23:03:39.548269 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:03:39.548247    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"797d8772836441cbbdc4f350a6e3f5d26e2a4e7f1277ad18460d955ff9d9248b\": container with ID starting with 797d8772836441cbbdc4f350a6e3f5d26e2a4e7f1277ad18460d955ff9d9248b not found: ID does not exist" containerID="797d8772836441cbbdc4f350a6e3f5d26e2a4e7f1277ad18460d955ff9d9248b"
Nov 03 23:03:39.548327 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:39.548277    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"797d8772836441cbbdc4f350a6e3f5d26e2a4e7f1277ad18460d955ff9d9248b"} err="failed to get container status \"797d8772836441cbbdc4f350a6e3f5d26e2a4e7f1277ad18460d955ff9d9248b\": rpc error: code = NotFound desc = could not find container \"797d8772836441cbbdc4f350a6e3f5d26e2a4e7f1277ad18460d955ff9d9248b\": container with ID starting with 797d8772836441cbbdc4f350a6e3f5d26e2a4e7f1277ad18460d955ff9d9248b not found: ID does not exist"
Nov 03 23:03:39.548327 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:39.548297    4431 scope.go:122] "RemoveContainer" containerID="4585bf084388c39eeaa35cf1e4f2a9043e63ba6917678722fb9841ed01fee112"
Nov 03 23:03:39.557330 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:39.557306    4431 scope.go:122] "RemoveContainer" containerID="4585bf084388c39eeaa35cf1e4f2a9043e63ba6917678722fb9841ed01fee112"
Nov 03 23:03:39.557609 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:03:39.557590    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4585bf084388c39eeaa35cf1e4f2a9043e63ba6917678722fb9841ed01fee112\": container with ID starting with 4585bf084388c39eeaa35cf1e4f2a9043e63ba6917678722fb9841ed01fee112 not found: ID does not exist" containerID="4585bf084388c39eeaa35cf1e4f2a9043e63ba6917678722fb9841ed01fee112"
Nov 03 23:03:39.557671 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:39.557618    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4585bf084388c39eeaa35cf1e4f2a9043e63ba6917678722fb9841ed01fee112"} err="failed to get container status \"4585bf084388c39eeaa35cf1e4f2a9043e63ba6917678722fb9841ed01fee112\": rpc error: code = NotFound desc = could not find container \"4585bf084388c39eeaa35cf1e4f2a9043e63ba6917678722fb9841ed01fee112\": container with ID starting with 4585bf084388c39eeaa35cf1e4f2a9043e63ba6917678722fb9841ed01fee112 not found: ID does not exist"
Nov 03 23:03:39.585360 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:39.585328    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:03:39.602513 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:39.602485    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fc68969-ebce-4cfb-bdb0-8600510e931f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fc68969-ebce-4cfb-bdb0-8600510e931f-model-cache\") pod \"0fc68969-ebce-4cfb-bdb0-8600510e931f\" (UID: \"0fc68969-ebce-4cfb-bdb0-8600510e931f\") "
Nov 03 23:03:39.602701 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:39.602567    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fc68969-ebce-4cfb-bdb0-8600510e931f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fc68969-ebce-4cfb-bdb0-8600510e931f-dshm\") pod \"0fc68969-ebce-4cfb-bdb0-8600510e931f\" (UID: \"0fc68969-ebce-4cfb-bdb0-8600510e931f\") "
Nov 03 23:03:39.602701 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:39.602625    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fc68969-ebce-4cfb-bdb0-8600510e931f-kube-api-access-tv25f\" (UniqueName: \"kubernetes.io/projected/0fc68969-ebce-4cfb-bdb0-8600510e931f-kube-api-access-tv25f\") pod \"0fc68969-ebce-4cfb-bdb0-8600510e931f\" (UID: \"0fc68969-ebce-4cfb-bdb0-8600510e931f\") "
Nov 03 23:03:39.602825 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:39.602725    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fc68969-ebce-4cfb-bdb0-8600510e931f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fc68969-ebce-4cfb-bdb0-8600510e931f-tmp-dir\") pod \"0fc68969-ebce-4cfb-bdb0-8600510e931f\" (UID: \"0fc68969-ebce-4cfb-bdb0-8600510e931f\") "
Nov 03 23:03:39.602825 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:39.602786    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fc68969-ebce-4cfb-bdb0-8600510e931f-model-cache" pod "0fc68969-ebce-4cfb-bdb0-8600510e931f" (UID: "0fc68969-ebce-4cfb-bdb0-8600510e931f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:03:39.602936 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:39.602843    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fc68969-ebce-4cfb-bdb0-8600510e931f-tls-certs\" (UniqueName: \"kubernetes.io/secret/0fc68969-ebce-4cfb-bdb0-8600510e931f-tls-certs\") pod \"0fc68969-ebce-4cfb-bdb0-8600510e931f\" (UID: \"0fc68969-ebce-4cfb-bdb0-8600510e931f\") "
Nov 03 23:03:39.602936 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:39.602884    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fc68969-ebce-4cfb-bdb0-8600510e931f-home\" (UniqueName: \"kubernetes.io/empty-dir/0fc68969-ebce-4cfb-bdb0-8600510e931f-home\") pod \"0fc68969-ebce-4cfb-bdb0-8600510e931f\" (UID: \"0fc68969-ebce-4cfb-bdb0-8600510e931f\") "
Nov 03 23:03:39.603359 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:39.603332    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fc68969-ebce-4cfb-bdb0-8600510e931f-tmp-dir" pod "0fc68969-ebce-4cfb-bdb0-8600510e931f" (UID: "0fc68969-ebce-4cfb-bdb0-8600510e931f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:03:39.603517 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:39.603486    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fc68969-ebce-4cfb-bdb0-8600510e931f-home" pod "0fc68969-ebce-4cfb-bdb0-8600510e931f" (UID: "0fc68969-ebce-4cfb-bdb0-8600510e931f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:03:39.603517 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:39.603511    4431 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fc68969-ebce-4cfb-bdb0-8600510e931f-tmp-dir\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:03:39.603670 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:39.603531    4431 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fc68969-ebce-4cfb-bdb0-8600510e931f-model-cache\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:03:39.605255 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:39.605203    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fc68969-ebce-4cfb-bdb0-8600510e931f-dshm" pod "0fc68969-ebce-4cfb-bdb0-8600510e931f" (UID: "0fc68969-ebce-4cfb-bdb0-8600510e931f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:03:39.605918 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:39.605900    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fc68969-ebce-4cfb-bdb0-8600510e931f-tls-certs" pod "0fc68969-ebce-4cfb-bdb0-8600510e931f" (UID: "0fc68969-ebce-4cfb-bdb0-8600510e931f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:03:39.606047 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:39.606008    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fc68969-ebce-4cfb-bdb0-8600510e931f-kube-api-access-tv25f" pod "0fc68969-ebce-4cfb-bdb0-8600510e931f" (UID: "0fc68969-ebce-4cfb-bdb0-8600510e931f"). InnerVolumeSpecName "kube-api-access-tv25f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:03:39.614271 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:39.614248    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d4b6bb-878757bb6-fxn94"]
Nov 03 23:03:39.618546 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:39.618522    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d4b6bb-878757bb6-fxn94"]
Nov 03 23:03:39.707670 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:39.706268    4431 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fc68969-ebce-4cfb-bdb0-8600510e931f-dshm\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:03:39.707670 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:39.706322    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tv25f\" (UniqueName: \"kubernetes.io/projected/0fc68969-ebce-4cfb-bdb0-8600510e931f-kube-api-access-tv25f\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:03:39.707670 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:39.706341    4431 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fc68969-ebce-4cfb-bdb0-8600510e931f-tls-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:03:39.707670 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:39.706359    4431 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fc68969-ebce-4cfb-bdb0-8600510e931f-home\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:03:39.891880 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:39.891844    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-3920cb/ipp-route-3920cb-kserve-76df875dbf-rrqjq"]
Nov 03 23:03:39.896381 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:39.896353    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-3920cb/ipp-route-3920cb-kserve-76df875dbf-rrqjq"]
Nov 03 23:03:40.026180 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:40.026139    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:03:40.092570 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:40.092537    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fc68969-ebce-4cfb-bdb0-8600510e931f" path="/var/lib/kubelet/pods/0fc68969-ebce-4cfb-bdb0-8600510e931f/volumes"
Nov 03 23:03:40.093017 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:40.093003    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb8f978d-2eb9-4cf4-a7ea-ca3caf0c11ab" path="/var/lib/kubelet/pods/bb8f978d-2eb9-4cf4-a7ea-ca3caf0c11ab/volumes"
Nov 03 23:03:41.348795 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:41.348754    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:03:42.431385 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:42.431354    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:03:42.746072 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:42.745945    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:03:42.869778 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:42.869745    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-162d7b-74ddd95c5-rpng8"]
Nov 03 23:03:42.870437 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:42.870419    4431 cpu_manager.go:414] "containerMap: removing container" podUID="bb8f978d-2eb9-4cf4-a7ea-ca3caf0c11ab" containerName="maas-api"
Nov 03 23:03:42.870521 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:42.870438    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb8f978d-2eb9-4cf4-a7ea-ca3caf0c11ab" containerName="maas-api"
Nov 03 23:03:42.870521 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:42.870473    4431 cpu_manager.go:414] "containerMap: removing container" podUID="0fc68969-ebce-4cfb-bdb0-8600510e931f" containerName="main"
Nov 03 23:03:42.870521 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:42.870478    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fc68969-ebce-4cfb-bdb0-8600510e931f" containerName="main"
Nov 03 23:03:42.870666 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:42.870575    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fc68969-ebce-4cfb-bdb0-8600510e931f" containerName="main"
Nov 03 23:03:42.870666 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:42.870589    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb8f978d-2eb9-4cf4-a7ea-ca3caf0c11ab" containerName="maas-api"
Nov 03 23:03:42.875789 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:42.875765    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-162d7b-74ddd95c5-rpng8"
Nov 03 23:03:42.901329 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:42.894099    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-162d7b-74ddd95c5-rpng8"]
Nov 03 23:03:42.901791 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:42.901765    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-162d7b\""
Nov 03 23:03:42.901953 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:42.901914    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-162d7b-dockercfg-26bg8\""
Nov 03 23:03:42.945310 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:42.945280    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72eb7f3b-8510-4699-8764-72c129b39943-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-162d7b-74ddd95c5-rpng8\" (UID: \"72eb7f3b-8510-4699-8764-72c129b39943\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-162d7b-74ddd95c5-rpng8"
Nov 03 23:03:42.945519 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:42.945498    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j85h\" (UniqueName: \"kubernetes.io/projected/72eb7f3b-8510-4699-8764-72c129b39943-kube-api-access-7j85h\") pod \"payload-pre-processing-e2e-ipp-b-162d7b-74ddd95c5-rpng8\" (UID: \"72eb7f3b-8510-4699-8764-72c129b39943\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-162d7b-74ddd95c5-rpng8"
Nov 03 23:03:43.006052 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:43.005970    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-162d7b-699f7476d4-tdvfk"]
Nov 03 23:03:43.010953 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:43.010930    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-162d7b-699f7476d4-tdvfk"
Nov 03 23:03:43.028029 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:43.027982    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-162d7b-699f7476d4-tdvfk"]
Nov 03 23:03:43.047553 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:43.047520    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j85h\" (UniqueName: \"kubernetes.io/projected/72eb7f3b-8510-4699-8764-72c129b39943-kube-api-access-7j85h\") pod \"payload-pre-processing-e2e-ipp-b-162d7b-74ddd95c5-rpng8\" (UID: \"72eb7f3b-8510-4699-8764-72c129b39943\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-162d7b-74ddd95c5-rpng8"
Nov 03 23:03:43.047717 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:43.047617    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj7rs\" (UniqueName: \"kubernetes.io/projected/61d2223e-d3b7-42ec-956d-ac7b3f94180c-kube-api-access-xj7rs\") pod \"payload-processing-e2e-ipp-b-162d7b-699f7476d4-tdvfk\" (UID: \"61d2223e-d3b7-42ec-956d-ac7b3f94180c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-162d7b-699f7476d4-tdvfk"
Nov 03 23:03:43.047797 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:43.047763    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72eb7f3b-8510-4699-8764-72c129b39943-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-162d7b-74ddd95c5-rpng8\" (UID: \"72eb7f3b-8510-4699-8764-72c129b39943\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-162d7b-74ddd95c5-rpng8"
Nov 03 23:03:43.047866 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:43.047848    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61d2223e-d3b7-42ec-956d-ac7b3f94180c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-162d7b-699f7476d4-tdvfk\" (UID: \"61d2223e-d3b7-42ec-956d-ac7b3f94180c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-162d7b-699f7476d4-tdvfk"
Nov 03 23:03:43.048483 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:43.048465    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72eb7f3b-8510-4699-8764-72c129b39943-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-162d7b-74ddd95c5-rpng8\" (UID: \"72eb7f3b-8510-4699-8764-72c129b39943\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-162d7b-74ddd95c5-rpng8"
Nov 03 23:03:43.070981 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:43.070950    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j85h\" (UniqueName: \"kubernetes.io/projected/72eb7f3b-8510-4699-8764-72c129b39943-kube-api-access-7j85h\") pod \"payload-pre-processing-e2e-ipp-b-162d7b-74ddd95c5-rpng8\" (UID: \"72eb7f3b-8510-4699-8764-72c129b39943\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-162d7b-74ddd95c5-rpng8"
Nov 03 23:03:43.148773 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:43.148737    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj7rs\" (UniqueName: \"kubernetes.io/projected/61d2223e-d3b7-42ec-956d-ac7b3f94180c-kube-api-access-xj7rs\") pod \"payload-processing-e2e-ipp-b-162d7b-699f7476d4-tdvfk\" (UID: \"61d2223e-d3b7-42ec-956d-ac7b3f94180c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-162d7b-699f7476d4-tdvfk"
Nov 03 23:03:43.148991 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:43.148955    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61d2223e-d3b7-42ec-956d-ac7b3f94180c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-162d7b-699f7476d4-tdvfk\" (UID: \"61d2223e-d3b7-42ec-956d-ac7b3f94180c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-162d7b-699f7476d4-tdvfk"
Nov 03 23:03:43.149766 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:43.149742    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61d2223e-d3b7-42ec-956d-ac7b3f94180c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-162d7b-699f7476d4-tdvfk\" (UID: \"61d2223e-d3b7-42ec-956d-ac7b3f94180c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-162d7b-699f7476d4-tdvfk"
Nov 03 23:03:43.171888 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:43.171864    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj7rs\" (UniqueName: \"kubernetes.io/projected/61d2223e-d3b7-42ec-956d-ac7b3f94180c-kube-api-access-xj7rs\") pod \"payload-processing-e2e-ipp-b-162d7b-699f7476d4-tdvfk\" (UID: \"61d2223e-d3b7-42ec-956d-ac7b3f94180c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-162d7b-699f7476d4-tdvfk"
Nov 03 23:03:43.203434 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:43.203397    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-162d7b-74ddd95c5-rpng8"
Nov 03 23:03:43.324890 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:43.324859    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-162d7b-699f7476d4-tdvfk"
Nov 03 23:03:43.510188 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:43.510161    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-162d7b-74ddd95c5-rpng8"]
Nov 03 23:03:43.511689 ip-10-0-139-73 kubenswrapper[4431]: W1103 23:03:43.511632    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72eb7f3b_8510_4699_8764_72c129b39943.slice/crio-1d0cb7c2af4eb39ef0d691acf67fec49db0a2ce371ad8853b4d954fa9855a653 WatchSource:0}: Error finding container 1d0cb7c2af4eb39ef0d691acf67fec49db0a2ce371ad8853b4d954fa9855a653: Status 404 returned error can't find the container with id 1d0cb7c2af4eb39ef0d691acf67fec49db0a2ce371ad8853b4d954fa9855a653
Nov 03 23:03:43.564175 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:43.564144    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:03:43.571757 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:43.571724    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-162d7b-74ddd95c5-rpng8" event={"ID":"72eb7f3b-8510-4699-8764-72c129b39943","Type":"ContainerStarted","Data":"1d0cb7c2af4eb39ef0d691acf67fec49db0a2ce371ad8853b4d954fa9855a653"}
Nov 03 23:03:43.629636 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:43.629612    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-162d7b-699f7476d4-tdvfk"]
Nov 03 23:03:43.630570 ip-10-0-139-73 kubenswrapper[4431]: W1103 23:03:43.630548    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61d2223e_d3b7_42ec_956d_ac7b3f94180c.slice/crio-ed511801f19085657cfc239dcc93f25c0e8f18c85e28501f7e6f8bd539c2d55f WatchSource:0}: Error finding container ed511801f19085657cfc239dcc93f25c0e8f18c85e28501f7e6f8bd539c2d55f: Status 404 returned error can't find the container with id ed511801f19085657cfc239dcc93f25c0e8f18c85e28501f7e6f8bd539c2d55f
Nov 03 23:03:43.678373 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:43.678347    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm"]
Nov 03 23:03:43.683585 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:43.683565    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm"
Nov 03 23:03:43.702928 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:43.702901    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj"]
Nov 03 23:03:43.709031 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:43.709004    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj"
Nov 03 23:03:43.722551 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:43.722484    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-4ad693-openshift-default-dockercfg-nvt7m\""
Nov 03 23:03:43.724870 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:43.724830    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm"]
Nov 03 23:03:43.742157 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:43.742053    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj"]
Nov 03 23:03:43.761900 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:43.761817    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acb636f9-9d36-49e8-af20-67b01ee4c095-istio-podinfo\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm\" (UID: \"acb636f9-9d36-49e8-af20-67b01ee4c095\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm"
Nov 03 23:03:43.761900 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:43.761885    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acb636f9-9d36-49e8-af20-67b01ee4c095-workload-certs\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm\" (UID: \"acb636f9-9d36-49e8-af20-67b01ee4c095\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm"
Nov 03 23:03:43.762132 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:43.761926    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5ba197f-88fc-484c-8fbd-b52d1b601a99-istio-token\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj\" (UID: \"e5ba197f-88fc-484c-8fbd-b52d1b601a99\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj"
Nov 03 23:03:43.762132 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:43.762059    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acb636f9-9d36-49e8-af20-67b01ee4c095-istio-token\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm\" (UID: \"acb636f9-9d36-49e8-af20-67b01ee4c095\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm"
Nov 03 23:03:43.762132 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:43.762095    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acb636f9-9d36-49e8-af20-67b01ee4c095-workload-socket\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm\" (UID: \"acb636f9-9d36-49e8-af20-67b01ee4c095\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm"
Nov 03 23:03:43.762409 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:43.762140    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5ba197f-88fc-484c-8fbd-b52d1b601a99-workload-certs\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj\" (UID: \"e5ba197f-88fc-484c-8fbd-b52d1b601a99\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj"
Nov 03 23:03:43.762409 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:43.762197    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5ba197f-88fc-484c-8fbd-b52d1b601a99-istio-data\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj\" (UID: \"e5ba197f-88fc-484c-8fbd-b52d1b601a99\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj"
Nov 03 23:03:43.762409 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:43.762271    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acb636f9-9d36-49e8-af20-67b01ee4c095-istio-envoy\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm\" (UID: \"acb636f9-9d36-49e8-af20-67b01ee4c095\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm"
Nov 03 23:03:43.762409 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:43.762311    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5ba197f-88fc-484c-8fbd-b52d1b601a99-istio-podinfo\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj\" (UID: \"e5ba197f-88fc-484c-8fbd-b52d1b601a99\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj"
Nov 03 23:03:43.762409 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:43.762368    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5ba197f-88fc-484c-8fbd-b52d1b601a99-istiod-ca-cert\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj\" (UID: \"e5ba197f-88fc-484c-8fbd-b52d1b601a99\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj"
Nov 03 23:03:43.762627 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:43.762409    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acb636f9-9d36-49e8-af20-67b01ee4c095-credential-socket\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm\" (UID: \"acb636f9-9d36-49e8-af20-67b01ee4c095\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm"
Nov 03 23:03:43.762627 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:43.762490    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47vq2\" (UniqueName: \"kubernetes.io/projected/e5ba197f-88fc-484c-8fbd-b52d1b601a99-kube-api-access-47vq2\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj\" (UID: \"e5ba197f-88fc-484c-8fbd-b52d1b601a99\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj"
Nov 03 23:03:43.762627 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:43.762549    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ba197f-88fc-484c-8fbd-b52d1b601a99-credential-socket\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj\" (UID: \"e5ba197f-88fc-484c-8fbd-b52d1b601a99\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj"
Nov 03 23:03:43.762722 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:43.762656    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp6m5\" (UniqueName: \"kubernetes.io/projected/acb636f9-9d36-49e8-af20-67b01ee4c095-kube-api-access-pp6m5\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm\" (UID: \"acb636f9-9d36-49e8-af20-67b01ee4c095\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm"
Nov 03 23:03:43.762757 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:43.762739    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5ba197f-88fc-484c-8fbd-b52d1b601a99-istio-envoy\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj\" (UID: \"e5ba197f-88fc-484c-8fbd-b52d1b601a99\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj"
Nov 03 23:03:43.762795 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:43.762776    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acb636f9-9d36-49e8-af20-67b01ee4c095-istio-data\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm\" (UID: \"acb636f9-9d36-49e8-af20-67b01ee4c095\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm"
Nov 03 23:03:43.765053 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:43.762925    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acb636f9-9d36-49e8-af20-67b01ee4c095-istiod-ca-cert\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm\" (UID: \"acb636f9-9d36-49e8-af20-67b01ee4c095\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm"
Nov 03 23:03:43.765053 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:43.763003    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ba197f-88fc-484c-8fbd-b52d1b601a99-workload-socket\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj\" (UID: \"e5ba197f-88fc-484c-8fbd-b52d1b601a99\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj"
Nov 03 23:03:43.866416 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:43.866377    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ba197f-88fc-484c-8fbd-b52d1b601a99-workload-socket\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj\" (UID: \"e5ba197f-88fc-484c-8fbd-b52d1b601a99\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj"
Nov 03 23:03:43.866615 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:43.866449    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acb636f9-9d36-49e8-af20-67b01ee4c095-istio-podinfo\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm\" (UID: \"acb636f9-9d36-49e8-af20-67b01ee4c095\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm"
Nov 03 23:03:43.866615 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:43.866476    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acb636f9-9d36-49e8-af20-67b01ee4c095-workload-certs\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm\" (UID: \"acb636f9-9d36-49e8-af20-67b01ee4c095\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm"
Nov 03 23:03:43.866615 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:43.866507    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5ba197f-88fc-484c-8fbd-b52d1b601a99-istio-token\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj\" (UID: \"e5ba197f-88fc-484c-8fbd-b52d1b601a99\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj"
Nov 03 23:03:43.866615 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:43.866533    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acb636f9-9d36-49e8-af20-67b01ee4c095-istio-token\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm\" (UID: \"acb636f9-9d36-49e8-af20-67b01ee4c095\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm"
Nov 03 23:03:43.866615 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:43.866558    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acb636f9-9d36-49e8-af20-67b01ee4c095-workload-socket\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm\" (UID: \"acb636f9-9d36-49e8-af20-67b01ee4c095\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm"
Nov 03 23:03:43.866615 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:43.866593    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5ba197f-88fc-484c-8fbd-b52d1b601a99-workload-certs\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj\" (UID: \"e5ba197f-88fc-484c-8fbd-b52d1b601a99\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj"
Nov 03 23:03:43.866956 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:43.866638    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5ba197f-88fc-484c-8fbd-b52d1b601a99-istio-data\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj\" (UID: \"e5ba197f-88fc-484c-8fbd-b52d1b601a99\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj"
Nov 03 23:03:43.866956 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:43.866673    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acb636f9-9d36-49e8-af20-67b01ee4c095-istio-envoy\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm\" (UID: \"acb636f9-9d36-49e8-af20-67b01ee4c095\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm"
Nov 03 23:03:43.866956 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:43.866710    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5ba197f-88fc-484c-8fbd-b52d1b601a99-istio-podinfo\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj\" (UID: \"e5ba197f-88fc-484c-8fbd-b52d1b601a99\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj"
Nov 03 23:03:43.866956 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:43.866760    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5ba197f-88fc-484c-8fbd-b52d1b601a99-istiod-ca-cert\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj\" (UID: \"e5ba197f-88fc-484c-8fbd-b52d1b601a99\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj"
Nov 03 23:03:43.866956 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:43.866794    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acb636f9-9d36-49e8-af20-67b01ee4c095-credential-socket\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm\" (UID: \"acb636f9-9d36-49e8-af20-67b01ee4c095\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm"
Nov 03 23:03:43.866956 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:43.866821    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ba197f-88fc-484c-8fbd-b52d1b601a99-workload-socket\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj\" (UID: \"e5ba197f-88fc-484c-8fbd-b52d1b601a99\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj"
Nov 03 23:03:43.866956 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:43.866847    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47vq2\" (UniqueName: \"kubernetes.io/projected/e5ba197f-88fc-484c-8fbd-b52d1b601a99-kube-api-access-47vq2\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj\" (UID: \"e5ba197f-88fc-484c-8fbd-b52d1b601a99\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj"
Nov 03 23:03:43.866956 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:43.866875    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ba197f-88fc-484c-8fbd-b52d1b601a99-credential-socket\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj\" (UID: \"e5ba197f-88fc-484c-8fbd-b52d1b601a99\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj"
Nov 03 23:03:43.866956 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:43.866941    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acb636f9-9d36-49e8-af20-67b01ee4c095-workload-certs\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm\" (UID: \"acb636f9-9d36-49e8-af20-67b01ee4c095\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm"
Nov 03 23:03:43.867470 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:43.866966    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp6m5\" (UniqueName: \"kubernetes.io/projected/acb636f9-9d36-49e8-af20-67b01ee4c095-kube-api-access-pp6m5\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm\" (UID: \"acb636f9-9d36-49e8-af20-67b01ee4c095\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm"
Nov 03 23:03:43.867470 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:43.867005    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5ba197f-88fc-484c-8fbd-b52d1b601a99-istio-envoy\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj\" (UID: \"e5ba197f-88fc-484c-8fbd-b52d1b601a99\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj"
Nov 03 23:03:43.867470 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:43.867031    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acb636f9-9d36-49e8-af20-67b01ee4c095-istio-data\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm\" (UID: \"acb636f9-9d36-49e8-af20-67b01ee4c095\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm"
Nov 03 23:03:43.867470 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:43.867073    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acb636f9-9d36-49e8-af20-67b01ee4c095-istiod-ca-cert\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm\" (UID: \"acb636f9-9d36-49e8-af20-67b01ee4c095\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm"
Nov 03 23:03:43.867470 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:43.867098    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acb636f9-9d36-49e8-af20-67b01ee4c095-workload-socket\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm\" (UID: \"acb636f9-9d36-49e8-af20-67b01ee4c095\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm"
Nov 03 23:03:43.867470 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:43.867098    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5ba197f-88fc-484c-8fbd-b52d1b601a99-istio-data\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj\" (UID: \"e5ba197f-88fc-484c-8fbd-b52d1b601a99\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj"
Nov 03 23:03:43.867811 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:43.867518    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acb636f9-9d36-49e8-af20-67b01ee4c095-credential-socket\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm\" (UID: \"acb636f9-9d36-49e8-af20-67b01ee4c095\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm"
Nov 03 23:03:43.867811 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:43.867532    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5ba197f-88fc-484c-8fbd-b52d1b601a99-workload-certs\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj\" (UID: \"e5ba197f-88fc-484c-8fbd-b52d1b601a99\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj"
Nov 03 23:03:43.867811 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:43.867686    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ba197f-88fc-484c-8fbd-b52d1b601a99-credential-socket\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj\" (UID: \"e5ba197f-88fc-484c-8fbd-b52d1b601a99\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj"
Nov 03 23:03:43.868829 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:43.868801    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5ba197f-88fc-484c-8fbd-b52d1b601a99-istiod-ca-cert\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj\" (UID: \"e5ba197f-88fc-484c-8fbd-b52d1b601a99\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj"
Nov 03 23:03:43.869012 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:43.868817    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acb636f9-9d36-49e8-af20-67b01ee4c095-istio-data\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm\" (UID: \"acb636f9-9d36-49e8-af20-67b01ee4c095\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm"
Nov 03 23:03:43.869208 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:43.869164    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acb636f9-9d36-49e8-af20-67b01ee4c095-istiod-ca-cert\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm\" (UID: \"acb636f9-9d36-49e8-af20-67b01ee4c095\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm"
Nov 03 23:03:43.869790 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:43.869761    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acb636f9-9d36-49e8-af20-67b01ee4c095-istio-podinfo\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm\" (UID: \"acb636f9-9d36-49e8-af20-67b01ee4c095\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm"
Nov 03 23:03:43.870136 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:43.870114    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5ba197f-88fc-484c-8fbd-b52d1b601a99-istio-podinfo\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj\" (UID: \"e5ba197f-88fc-484c-8fbd-b52d1b601a99\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj"
Nov 03 23:03:43.870334 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:43.870316    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acb636f9-9d36-49e8-af20-67b01ee4c095-istio-envoy\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm\" (UID: \"acb636f9-9d36-49e8-af20-67b01ee4c095\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm"
Nov 03 23:03:43.870707 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:43.870689    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5ba197f-88fc-484c-8fbd-b52d1b601a99-istio-envoy\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj\" (UID: \"e5ba197f-88fc-484c-8fbd-b52d1b601a99\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj"
Nov 03 23:03:43.892629 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:43.892602    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acb636f9-9d36-49e8-af20-67b01ee4c095-istio-token\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm\" (UID: \"acb636f9-9d36-49e8-af20-67b01ee4c095\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm"
Nov 03 23:03:43.892629 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:43.892612    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp6m5\" (UniqueName: \"kubernetes.io/projected/acb636f9-9d36-49e8-af20-67b01ee4c095-kube-api-access-pp6m5\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm\" (UID: \"acb636f9-9d36-49e8-af20-67b01ee4c095\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm"
Nov 03 23:03:43.895142 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:43.895122    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5ba197f-88fc-484c-8fbd-b52d1b601a99-istio-token\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj\" (UID: \"e5ba197f-88fc-484c-8fbd-b52d1b601a99\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj"
Nov 03 23:03:43.896755 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:43.896728    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47vq2\" (UniqueName: \"kubernetes.io/projected/e5ba197f-88fc-484c-8fbd-b52d1b601a99-kube-api-access-47vq2\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj\" (UID: \"e5ba197f-88fc-484c-8fbd-b52d1b601a99\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj"
Nov 03 23:03:43.998470 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:43.998437    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm"
Nov 03 23:03:44.074401 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:44.074329    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj"
Nov 03 23:03:44.340348 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:44.340318    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm"]
Nov 03 23:03:44.341491 ip-10-0-139-73 kubenswrapper[4431]: W1103 23:03:44.341465    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacb636f9_9d36_49e8_af20_67b01ee4c095.slice/crio-4f3c746ea0cd262b5b56ec645c873c70432cd1f7a44713e364c2c231e163138d WatchSource:0}: Error finding container 4f3c746ea0cd262b5b56ec645c873c70432cd1f7a44713e364c2c231e163138d: Status 404 returned error can't find the container with id 4f3c746ea0cd262b5b56ec645c873c70432cd1f7a44713e364c2c231e163138d
Nov 03 23:03:44.345207 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:44.345163    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:03:44.345324 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:44.345269    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:03:44.345324 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:44.345312    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:03:44.446016 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:44.445987    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj"]
Nov 03 23:03:44.446729 ip-10-0-139-73 kubenswrapper[4431]: W1103 23:03:44.446707    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5ba197f_88fc_484c_8fbd_b52d1b601a99.slice/crio-f6188fdfda650dd1396a55bcecbc5d84aace305e8f0cd7a9bb86a91037718ef6 WatchSource:0}: Error finding container f6188fdfda650dd1396a55bcecbc5d84aace305e8f0cd7a9bb86a91037718ef6: Status 404 returned error can't find the container with id f6188fdfda650dd1396a55bcecbc5d84aace305e8f0cd7a9bb86a91037718ef6
Nov 03 23:03:44.450642 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:44.450603    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:03:44.450731 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:44.450700    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:03:44.450778 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:44.450762    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:03:44.584259 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:44.583726    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-162d7b-699f7476d4-tdvfk" event={"ID":"61d2223e-d3b7-42ec-956d-ac7b3f94180c","Type":"ContainerStarted","Data":"cc1980887317183449b97da7603ad579393d8e518a3c549586fdc49a697ce009"}
Nov 03 23:03:44.584259 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:44.584104    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-162d7b-699f7476d4-tdvfk"
Nov 03 23:03:44.584259 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:44.584134    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-162d7b-699f7476d4-tdvfk" event={"ID":"61d2223e-d3b7-42ec-956d-ac7b3f94180c","Type":"ContainerStarted","Data":"ed511801f19085657cfc239dcc93f25c0e8f18c85e28501f7e6f8bd539c2d55f"}
Nov 03 23:03:44.586672 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:44.586573    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm" event={"ID":"acb636f9-9d36-49e8-af20-67b01ee4c095","Type":"ContainerStarted","Data":"deed00d61b410ef18d2e7a55f69970ccedd03767e6a753b3eae9534c40ecdb03"}
Nov 03 23:03:44.586672 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:44.586631    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm" event={"ID":"acb636f9-9d36-49e8-af20-67b01ee4c095","Type":"ContainerStarted","Data":"4f3c746ea0cd262b5b56ec645c873c70432cd1f7a44713e364c2c231e163138d"}
Nov 03 23:03:44.589107 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:44.589047    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj" event={"ID":"e5ba197f-88fc-484c-8fbd-b52d1b601a99","Type":"ContainerStarted","Data":"3599fa82d00f100f265610a263318454b15f740dcbbbb7d563c97c5b2e3a0983"}
Nov 03 23:03:44.589304 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:44.589286    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj" event={"ID":"e5ba197f-88fc-484c-8fbd-b52d1b601a99","Type":"ContainerStarted","Data":"f6188fdfda650dd1396a55bcecbc5d84aace305e8f0cd7a9bb86a91037718ef6"}
Nov 03 23:03:44.590811 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:44.590787    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-162d7b-74ddd95c5-rpng8" event={"ID":"72eb7f3b-8510-4699-8764-72c129b39943","Type":"ContainerStarted","Data":"92b2c77138f684a8e9a52c48b23c5b1ae00c2521b4ddf0e79b3bc01c12aed7e9"}
Nov 03 23:03:44.591210 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:44.591193    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-162d7b-74ddd95c5-rpng8"
Nov 03 23:03:44.675270 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:44.673641    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm" podStartSLOduration=1.673621318 podStartE2EDuration="1.673621318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:03:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:03:44.671495327 +0000 UTC m=+3048.697351203" watchObservedRunningTime="2026-11-03 23:03:44.673621318 +0000 UTC m=+3048.699477181"
Nov 03 23:03:44.675804 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:44.675755    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-162d7b-699f7476d4-tdvfk" podStartSLOduration=2.675742258 podStartE2EDuration="2.675742258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:03:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:03:44.62604128 +0000 UTC m=+3048.651897155" watchObservedRunningTime="2026-11-03 23:03:44.675742258 +0000 UTC m=+3048.701598120"
Nov 03 23:03:44.715487 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:44.715426    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-162d7b-74ddd95c5-rpng8" podStartSLOduration=2.715412562 podStartE2EDuration="2.715412562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:03:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:03:44.711350272 +0000 UTC m=+3048.737206137" watchObservedRunningTime="2026-11-03 23:03:44.715412562 +0000 UTC m=+3048.741268423"
Nov 03 23:03:44.761858 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:44.761791    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj" podStartSLOduration=1.761771262 podStartE2EDuration="1.761771262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:03:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:03:44.756018788 +0000 UTC m=+3048.781874702" watchObservedRunningTime="2026-11-03 23:03:44.761771262 +0000 UTC m=+3048.787627124"
Nov 03 23:03:45.002241 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:45.001628    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm"
Nov 03 23:03:45.069369 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:45.006338    4431 patch_prober.go:29] interesting pod/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Nov 03 23:03:45.069369 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:45.006401    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm" podUID="acb636f9-9d36-49e8-af20-67b01ee4c095" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Nov 03 23:03:45.074561 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:45.074529    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj"
Nov 03 23:03:45.076175 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:45.076147    4431 patch_prober.go:29] interesting pod/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body=
Nov 03 23:03:45.076327 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:45.076194    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj" podUID="e5ba197f-88fc-484c-8fbd-b52d1b601a99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused"
Nov 03 23:03:45.224569 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:45.224530    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:03:45.513032 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:45.512997    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ad693-589cf5d8f6-lgfqh"
Nov 03 23:03:45.513261 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:45.513071    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ad693-5685dfd7bc-vbh2d"
Nov 03 23:03:45.999143 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:45.999101    4431 patch_prober.go:29] interesting pod/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Nov 03 23:03:45.999567 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:45.999172    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm" podUID="acb636f9-9d36-49e8-af20-67b01ee4c095" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Nov 03 23:03:46.078240 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:46.077900    4431 patch_prober.go:29] interesting pod/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body=
Nov 03 23:03:46.078240 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:46.077956    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj" podUID="e5ba197f-88fc-484c-8fbd-b52d1b601a99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused"
Nov 03 23:03:46.362298 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:46.358331    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:03:46.806916 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:46.806868    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:03:46.960241 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:46.958111    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:03:47.000270 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:46.999585    4431 patch_prober.go:29] interesting pod/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Nov 03 23:03:47.000270 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:46.999662    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm" podUID="acb636f9-9d36-49e8-af20-67b01ee4c095" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Nov 03 23:03:47.269254 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:47.266745    4431 patch_prober.go:29] interesting pod/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body=
Nov 03 23:03:47.269254 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:47.266812    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj" podUID="e5ba197f-88fc-484c-8fbd-b52d1b601a99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused"
Nov 03 23:03:47.280295 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:47.272300    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:03:47.309245 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:47.282650    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8"]
Nov 03 23:03:47.309245 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:47.282952    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8" podUID="4c62c97b-1d80-4227-a469-08c2fa89b868" containerName="istio-proxy" containerID="cri-o://6d971e3ff060b1f0171d344da36a60b908f6dcd39c4857e163dbad14cba84649" gracePeriod=30
Nov 03 23:03:47.416227 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:47.416179    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:03:47.576518 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:47.576420    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:03:47.998842 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:47.998800    4431 patch_prober.go:29] interesting pod/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Nov 03 23:03:47.999015 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:47.998860    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm" podUID="acb636f9-9d36-49e8-af20-67b01ee4c095" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Nov 03 23:03:48.099463 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:48.099424    4431 patch_prober.go:29] interesting pod/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body=
Nov 03 23:03:48.099964 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:48.099485    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj" podUID="e5ba197f-88fc-484c-8fbd-b52d1b601a99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused"
Nov 03 23:03:49.002241 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:49.000717    4431 patch_prober.go:29] interesting pod/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Nov 03 23:03:49.002241 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:49.000796    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm" podUID="acb636f9-9d36-49e8-af20-67b01ee4c095" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Nov 03 23:03:49.009884 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:49.009852    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:03:49.076604 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:49.076564    4431 patch_prober.go:29] interesting pod/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body=
Nov 03 23:03:49.076909 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:49.076875    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj" podUID="e5ba197f-88fc-484c-8fbd-b52d1b601a99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused"
Nov 03 23:03:50.001990 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:50.001679    4431 patch_prober.go:29] interesting pod/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Nov 03 23:03:50.001990 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:50.001739    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm" podUID="acb636f9-9d36-49e8-af20-67b01ee4c095" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Nov 03 23:03:50.081241 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:50.080343    4431 patch_prober.go:29] interesting pod/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body=
Nov 03 23:03:50.081241 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:50.080406    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj" podUID="e5ba197f-88fc-484c-8fbd-b52d1b601a99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused"
Nov 03 23:03:50.492521 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:50.492491    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:03:50.884875 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:50.884834    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:03:51.005921 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:51.005886    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm"
Nov 03 23:03:51.006444 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:51.006393    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm"
Nov 03 23:03:51.007225 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:51.007188    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm"
Nov 03 23:03:51.095280 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:51.095248    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj"
Nov 03 23:03:51.095646 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:51.095615    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj"
Nov 03 23:03:51.097002 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:51.096977    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj"
Nov 03 23:03:51.243410 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:51.241663    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:03:51.425909 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:51.425869    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:03:51.622075 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:51.621995    4431 patch_prober.go:29] interesting pod/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 23:03:51.622258 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:51.622074    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8" podUID="4c62c97b-1d80-4227-a469-08c2fa89b868" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 23:03:52.578554 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:52.578524    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8"
Nov 03 23:03:52.685737 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:52.685707    4431 generic.go:356] "Generic (PLEG): container finished" podID="4c62c97b-1d80-4227-a469-08c2fa89b868" containerID="6d971e3ff060b1f0171d344da36a60b908f6dcd39c4857e163dbad14cba84649" exitCode=0
Nov 03 23:03:52.685967 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:52.685760    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8"
Nov 03 23:03:52.685967 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:52.685789    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8" event={"ID":"4c62c97b-1d80-4227-a469-08c2fa89b868","Type":"ContainerDied","Data":"6d971e3ff060b1f0171d344da36a60b908f6dcd39c4857e163dbad14cba84649"}
Nov 03 23:03:52.685967 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:52.685858    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8" event={"ID":"4c62c97b-1d80-4227-a469-08c2fa89b868","Type":"ContainerDied","Data":"3e502a6b934f6211938d72064f99b5c9b79b4eada7e879f898a8a41cd3ebf873"}
Nov 03 23:03:52.685967 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:52.685892    4431 scope.go:122] "RemoveContainer" containerID="6d971e3ff060b1f0171d344da36a60b908f6dcd39c4857e163dbad14cba84649"
Nov 03 23:03:52.708611 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:52.707653    4431 scope.go:122] "RemoveContainer" containerID="6d971e3ff060b1f0171d344da36a60b908f6dcd39c4857e163dbad14cba84649"
Nov 03 23:03:52.709059 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:03:52.709026    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d971e3ff060b1f0171d344da36a60b908f6dcd39c4857e163dbad14cba84649\": container with ID starting with 6d971e3ff060b1f0171d344da36a60b908f6dcd39c4857e163dbad14cba84649 not found: ID does not exist" containerID="6d971e3ff060b1f0171d344da36a60b908f6dcd39c4857e163dbad14cba84649"
Nov 03 23:03:52.711648 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:52.709068    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d971e3ff060b1f0171d344da36a60b908f6dcd39c4857e163dbad14cba84649"} err="failed to get container status \"6d971e3ff060b1f0171d344da36a60b908f6dcd39c4857e163dbad14cba84649\": rpc error: code = NotFound desc = could not find container \"6d971e3ff060b1f0171d344da36a60b908f6dcd39c4857e163dbad14cba84649\": container with ID starting with 6d971e3ff060b1f0171d344da36a60b908f6dcd39c4857e163dbad14cba84649 not found: ID does not exist"
Nov 03 23:03:52.711648 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:52.709585    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:03:52.718207 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:52.718173    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c62c97b-1d80-4227-a469-08c2fa89b868-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c62c97b-1d80-4227-a469-08c2fa89b868-workload-certs\") pod \"4c62c97b-1d80-4227-a469-08c2fa89b868\" (UID: \"4c62c97b-1d80-4227-a469-08c2fa89b868\") "
Nov 03 23:03:52.718352 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:52.718246    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c62c97b-1d80-4227-a469-08c2fa89b868-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c62c97b-1d80-4227-a469-08c2fa89b868-istiod-ca-cert\") pod \"4c62c97b-1d80-4227-a469-08c2fa89b868\" (UID: \"4c62c97b-1d80-4227-a469-08c2fa89b868\") "
Nov 03 23:03:52.718352 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:52.718266    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4c62c97b-1d80-4227-a469-08c2fa89b868-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c62c97b-1d80-4227-a469-08c2fa89b868-istio-podinfo\") pod \"4c62c97b-1d80-4227-a469-08c2fa89b868\" (UID: \"4c62c97b-1d80-4227-a469-08c2fa89b868\") "
Nov 03 23:03:52.718352 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:52.718289    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c62c97b-1d80-4227-a469-08c2fa89b868-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c62c97b-1d80-4227-a469-08c2fa89b868-credential-socket\") pod \"4c62c97b-1d80-4227-a469-08c2fa89b868\" (UID: \"4c62c97b-1d80-4227-a469-08c2fa89b868\") "
Nov 03 23:03:52.718502 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:52.718360    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c62c97b-1d80-4227-a469-08c2fa89b868-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c62c97b-1d80-4227-a469-08c2fa89b868-workload-socket\") pod \"4c62c97b-1d80-4227-a469-08c2fa89b868\" (UID: \"4c62c97b-1d80-4227-a469-08c2fa89b868\") "
Nov 03 23:03:52.718502 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:52.718430    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c62c97b-1d80-4227-a469-08c2fa89b868-kube-api-access-zqpns\" (UniqueName: \"kubernetes.io/projected/4c62c97b-1d80-4227-a469-08c2fa89b868-kube-api-access-zqpns\") pod \"4c62c97b-1d80-4227-a469-08c2fa89b868\" (UID: \"4c62c97b-1d80-4227-a469-08c2fa89b868\") "
Nov 03 23:03:52.718502 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:52.718465    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c62c97b-1d80-4227-a469-08c2fa89b868-workload-certs" pod "4c62c97b-1d80-4227-a469-08c2fa89b868" (UID: "4c62c97b-1d80-4227-a469-08c2fa89b868"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:03:52.718670 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:52.718510    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c62c97b-1d80-4227-a469-08c2fa89b868-istio-token\" (UniqueName: \"kubernetes.io/projected/4c62c97b-1d80-4227-a469-08c2fa89b868-istio-token\") pod \"4c62c97b-1d80-4227-a469-08c2fa89b868\" (UID: \"4c62c97b-1d80-4227-a469-08c2fa89b868\") "
Nov 03 23:03:52.718670 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:52.718591    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c62c97b-1d80-4227-a469-08c2fa89b868-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c62c97b-1d80-4227-a469-08c2fa89b868-istio-data\") pod \"4c62c97b-1d80-4227-a469-08c2fa89b868\" (UID: \"4c62c97b-1d80-4227-a469-08c2fa89b868\") "
Nov 03 23:03:52.718670 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:52.718653    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c62c97b-1d80-4227-a469-08c2fa89b868-workload-socket" pod "4c62c97b-1d80-4227-a469-08c2fa89b868" (UID: "4c62c97b-1d80-4227-a469-08c2fa89b868"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:03:52.718837 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:52.718695    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c62c97b-1d80-4227-a469-08c2fa89b868-istiod-ca-cert" pod "4c62c97b-1d80-4227-a469-08c2fa89b868" (UID: "4c62c97b-1d80-4227-a469-08c2fa89b868"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:03:52.718837 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:52.718797    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c62c97b-1d80-4227-a469-08c2fa89b868-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c62c97b-1d80-4227-a469-08c2fa89b868-istio-envoy\") pod \"4c62c97b-1d80-4227-a469-08c2fa89b868\" (UID: \"4c62c97b-1d80-4227-a469-08c2fa89b868\") "
Nov 03 23:03:52.718954 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:52.718874    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c62c97b-1d80-4227-a469-08c2fa89b868-istio-data" pod "4c62c97b-1d80-4227-a469-08c2fa89b868" (UID: "4c62c97b-1d80-4227-a469-08c2fa89b868"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:03:52.719041 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:52.719017    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c62c97b-1d80-4227-a469-08c2fa89b868-credential-socket" pod "4c62c97b-1d80-4227-a469-08c2fa89b868" (UID: "4c62c97b-1d80-4227-a469-08c2fa89b868"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:03:52.719506 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:52.719486    4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c62c97b-1d80-4227-a469-08c2fa89b868-workload-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:03:52.719599 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:52.719509    4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c62c97b-1d80-4227-a469-08c2fa89b868-istiod-ca-cert\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:03:52.719599 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:52.719522    4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c62c97b-1d80-4227-a469-08c2fa89b868-credential-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:03:52.719599 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:52.719534    4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c62c97b-1d80-4227-a469-08c2fa89b868-workload-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:03:52.719599 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:52.719547    4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c62c97b-1d80-4227-a469-08c2fa89b868-istio-data\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:03:52.720642 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:52.720614    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4c62c97b-1d80-4227-a469-08c2fa89b868-istio-podinfo" pod "4c62c97b-1d80-4227-a469-08c2fa89b868" (UID: "4c62c97b-1d80-4227-a469-08c2fa89b868"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 23:03:52.720801 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:52.720784    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c62c97b-1d80-4227-a469-08c2fa89b868-istio-token" pod "4c62c97b-1d80-4227-a469-08c2fa89b868" (UID: "4c62c97b-1d80-4227-a469-08c2fa89b868"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:03:52.721078 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:52.721062    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c62c97b-1d80-4227-a469-08c2fa89b868-istio-envoy" pod "4c62c97b-1d80-4227-a469-08c2fa89b868" (UID: "4c62c97b-1d80-4227-a469-08c2fa89b868"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:03:52.721124 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:52.721101    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c62c97b-1d80-4227-a469-08c2fa89b868-kube-api-access-zqpns" pod "4c62c97b-1d80-4227-a469-08c2fa89b868" (UID: "4c62c97b-1d80-4227-a469-08c2fa89b868"). InnerVolumeSpecName "kube-api-access-zqpns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:03:52.821466 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:52.821422    4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c62c97b-1d80-4227-a469-08c2fa89b868-istio-token\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:03:52.821466 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:52.821453    4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c62c97b-1d80-4227-a469-08c2fa89b868-istio-envoy\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:03:52.821466 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:52.821463    4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c62c97b-1d80-4227-a469-08c2fa89b868-istio-podinfo\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:03:52.821466 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:52.821471    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqpns\" (UniqueName: \"kubernetes.io/projected/4c62c97b-1d80-4227-a469-08c2fa89b868-kube-api-access-zqpns\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:03:52.952128 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:52.952048    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:03:53.032907 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:53.032876    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8"]
Nov 03 23:03:53.042362 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:53.042327    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-d4b6bb-openshift-default-788fd77bff-8l4w8"]
Nov 03 23:03:53.051410 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:53.051385    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f6dcc0-85d8fff6db-tvftq"]
Nov 03 23:03:53.052005 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:53.051984    4431 cpu_manager.go:414] "containerMap: removing container" podUID="4c62c97b-1d80-4227-a469-08c2fa89b868" containerName="istio-proxy"
Nov 03 23:03:53.052059 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:53.052006    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c62c97b-1d80-4227-a469-08c2fa89b868" containerName="istio-proxy"
Nov 03 23:03:53.052124 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:53.052115    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c62c97b-1d80-4227-a469-08c2fa89b868" containerName="istio-proxy"
Nov 03 23:03:53.057148 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:53.056968    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f6dcc0-85d8fff6db-tvftq"
Nov 03 23:03:53.076718 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:53.076684    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f6dcc0-85d8fff6db-tvftq"]
Nov 03 23:03:53.077114 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:53.077089    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-f6dcc0\""
Nov 03 23:03:53.106255 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:53.106210    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f6dcc0-84b547ffbc-6v2rf"]
Nov 03 23:03:53.110392 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:53.110374    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f6dcc0-84b547ffbc-6v2rf"
Nov 03 23:03:53.117288 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:53.117260    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:03:53.129490 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:53.127346    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws724\" (UniqueName: \"kubernetes.io/projected/9b7dd4de-ba44-4b67-a140-55d8008f5c6f-kube-api-access-ws724\") pod \"payload-processing-e2e-ipp-b-f6dcc0-85d8fff6db-tvftq\" (UID: \"9b7dd4de-ba44-4b67-a140-55d8008f5c6f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f6dcc0-85d8fff6db-tvftq"
Nov 03 23:03:53.129490 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:53.127468    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b7dd4de-ba44-4b67-a140-55d8008f5c6f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f6dcc0-85d8fff6db-tvftq\" (UID: \"9b7dd4de-ba44-4b67-a140-55d8008f5c6f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f6dcc0-85d8fff6db-tvftq"
Nov 03 23:03:53.129490 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:53.127812    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f6dcc0-84b547ffbc-6v2rf"]
Nov 03 23:03:53.129490 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:53.128046    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-f6dcc0-dockercfg-f7lz9\""
Nov 03 23:03:53.240321 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:53.240134    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dea78d3-8945-4582-8fdb-38b0a28ad416-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f6dcc0-84b547ffbc-6v2rf\" (UID: \"8dea78d3-8945-4582-8fdb-38b0a28ad416\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f6dcc0-84b547ffbc-6v2rf"
Nov 03 23:03:53.240321 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:53.240302    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws724\" (UniqueName: \"kubernetes.io/projected/9b7dd4de-ba44-4b67-a140-55d8008f5c6f-kube-api-access-ws724\") pod \"payload-processing-e2e-ipp-b-f6dcc0-85d8fff6db-tvftq\" (UID: \"9b7dd4de-ba44-4b67-a140-55d8008f5c6f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f6dcc0-85d8fff6db-tvftq"
Nov 03 23:03:53.240556 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:53.240458    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b7dd4de-ba44-4b67-a140-55d8008f5c6f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f6dcc0-85d8fff6db-tvftq\" (UID: \"9b7dd4de-ba44-4b67-a140-55d8008f5c6f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f6dcc0-85d8fff6db-tvftq"
Nov 03 23:03:53.240556 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:53.240528    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mclqt\" (UniqueName: \"kubernetes.io/projected/8dea78d3-8945-4582-8fdb-38b0a28ad416-kube-api-access-mclqt\") pod \"payload-pre-processing-e2e-ipp-b-f6dcc0-84b547ffbc-6v2rf\" (UID: \"8dea78d3-8945-4582-8fdb-38b0a28ad416\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f6dcc0-84b547ffbc-6v2rf"
Nov 03 23:03:53.243754 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:53.242405    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b7dd4de-ba44-4b67-a140-55d8008f5c6f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f6dcc0-85d8fff6db-tvftq\" (UID: \"9b7dd4de-ba44-4b67-a140-55d8008f5c6f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f6dcc0-85d8fff6db-tvftq"
Nov 03 23:03:53.267089 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:53.267060    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws724\" (UniqueName: \"kubernetes.io/projected/9b7dd4de-ba44-4b67-a140-55d8008f5c6f-kube-api-access-ws724\") pod \"payload-processing-e2e-ipp-b-f6dcc0-85d8fff6db-tvftq\" (UID: \"9b7dd4de-ba44-4b67-a140-55d8008f5c6f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f6dcc0-85d8fff6db-tvftq"
Nov 03 23:03:53.341946 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:53.341914    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mclqt\" (UniqueName: \"kubernetes.io/projected/8dea78d3-8945-4582-8fdb-38b0a28ad416-kube-api-access-mclqt\") pod \"payload-pre-processing-e2e-ipp-b-f6dcc0-84b547ffbc-6v2rf\" (UID: \"8dea78d3-8945-4582-8fdb-38b0a28ad416\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f6dcc0-84b547ffbc-6v2rf"
Nov 03 23:03:53.342121 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:53.342001    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dea78d3-8945-4582-8fdb-38b0a28ad416-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f6dcc0-84b547ffbc-6v2rf\" (UID: \"8dea78d3-8945-4582-8fdb-38b0a28ad416\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f6dcc0-84b547ffbc-6v2rf"
Nov 03 23:03:53.342717 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:53.342699    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dea78d3-8945-4582-8fdb-38b0a28ad416-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f6dcc0-84b547ffbc-6v2rf\" (UID: \"8dea78d3-8945-4582-8fdb-38b0a28ad416\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f6dcc0-84b547ffbc-6v2rf"
Nov 03 23:03:53.364858 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:53.364831    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mclqt\" (UniqueName: \"kubernetes.io/projected/8dea78d3-8945-4582-8fdb-38b0a28ad416-kube-api-access-mclqt\") pod \"payload-pre-processing-e2e-ipp-b-f6dcc0-84b547ffbc-6v2rf\" (UID: \"8dea78d3-8945-4582-8fdb-38b0a28ad416\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f6dcc0-84b547ffbc-6v2rf"
Nov 03 23:03:53.373680 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:53.373654    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f6dcc0-85d8fff6db-tvftq"
Nov 03 23:03:53.431777 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:53.431749    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f6dcc0-84b547ffbc-6v2rf"
Nov 03 23:03:53.674443 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:53.674417    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f6dcc0-85d8fff6db-tvftq"]
Nov 03 23:03:53.676541 ip-10-0-139-73 kubenswrapper[4431]: W1103 23:03:53.676516    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b7dd4de_ba44_4b67_a140_55d8008f5c6f.slice/crio-911190bf90088e0869b0fc88c449adcfb07cd9b2c4f476250adb028bdd526013 WatchSource:0}: Error finding container 911190bf90088e0869b0fc88c449adcfb07cd9b2c4f476250adb028bdd526013: Status 404 returned error can't find the container with id 911190bf90088e0869b0fc88c449adcfb07cd9b2c4f476250adb028bdd526013
Nov 03 23:03:53.695397 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:53.695367    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f6dcc0-85d8fff6db-tvftq" event={"ID":"9b7dd4de-ba44-4b67-a140-55d8008f5c6f","Type":"ContainerStarted","Data":"911190bf90088e0869b0fc88c449adcfb07cd9b2c4f476250adb028bdd526013"}
Nov 03 23:03:53.769791 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:53.769393    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f6dcc0-84b547ffbc-6v2rf"]
Nov 03 23:03:53.773287 ip-10-0-139-73 kubenswrapper[4431]: W1103 23:03:53.773259    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dea78d3_8945_4582_8fdb_38b0a28ad416.slice/crio-24252ec472fe4c53f9213534e99127843e2819dc6a86902c4fb887abc1a4d795 WatchSource:0}: Error finding container 24252ec472fe4c53f9213534e99127843e2819dc6a86902c4fb887abc1a4d795: Status 404 returned error can't find the container with id 24252ec472fe4c53f9213534e99127843e2819dc6a86902c4fb887abc1a4d795
Nov 03 23:03:54.088828 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:54.088752    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c62c97b-1d80-4227-a469-08c2fa89b868" path="/var/lib/kubelet/pods/4c62c97b-1d80-4227-a469-08c2fa89b868/volumes"
Nov 03 23:03:54.492778 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:54.492738    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:03:54.707750 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:54.707713    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f6dcc0-84b547ffbc-6v2rf" event={"ID":"8dea78d3-8945-4582-8fdb-38b0a28ad416","Type":"ContainerStarted","Data":"83617f2d69af0616a4cc35ea4634b168c338ae955dabfb655387bfce9cfc4343"}
Nov 03 23:03:54.708256 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:54.707798    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f6dcc0-84b547ffbc-6v2rf" event={"ID":"8dea78d3-8945-4582-8fdb-38b0a28ad416","Type":"ContainerStarted","Data":"24252ec472fe4c53f9213534e99127843e2819dc6a86902c4fb887abc1a4d795"}
Nov 03 23:03:54.708256 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:54.707832    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f6dcc0-84b547ffbc-6v2rf"
Nov 03 23:03:54.709601 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:54.709576    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f6dcc0-85d8fff6db-tvftq" event={"ID":"9b7dd4de-ba44-4b67-a140-55d8008f5c6f","Type":"ContainerStarted","Data":"d556d32c2863bdbe5bc5c9c768764ec31152937133c520c17a780bc7850edcc1"}
Nov 03 23:03:54.709685 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:54.709630    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f6dcc0-85d8fff6db-tvftq"
Nov 03 23:03:54.750005 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:54.749912    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f6dcc0-84b547ffbc-6v2rf" podStartSLOduration=1.749896159 podStartE2EDuration="1.749896159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:03:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:03:54.74705989 +0000 UTC m=+3058.772915753" watchObservedRunningTime="2026-11-03 23:03:54.749896159 +0000 UTC m=+3058.775752021"
Nov 03 23:03:54.787347 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:54.787295    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-f6dcc0-85d8fff6db-tvftq" podStartSLOduration=1.787277087 podStartE2EDuration="1.787277087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:03:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:03:54.785696612 +0000 UTC m=+3058.811552474" watchObservedRunningTime="2026-11-03 23:03:54.787277087 +0000 UTC m=+3058.813132949"
Nov 03 23:03:55.529573 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:55.529538    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:03:55.620371 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:55.620343    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-162d7b-699f7476d4-tdvfk"
Nov 03 23:03:56.098210 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:56.098180    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:03:56.625394 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:56.625364    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-162d7b-74ddd95c5-rpng8"
Nov 03 23:03:57.628013 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:57.627966    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:03:58.035513 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:58.035478    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-4f9bb0-58c4b77cfb-cdgpk"]
Nov 03 23:03:58.067584 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:58.067551    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4f9bb0-6f8b69d999-g9rgk"]
Nov 03 23:03:58.210555 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:58.210521    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:03:58.349995 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:58.349908    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:03:59.149009 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:03:59.148974    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:04:00.899774 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:00.899732    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:04:01.262660 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:04:01.262622    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 23:04:01.671138 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:01.671102    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:04:02.219760 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:02.219708    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:04:02.781032 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:02.780998    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:04:03.035808 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:03.035725    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4f9bb0-58c4b77cfb-cdgpk" podUID="96145989-02fa-41e0-9a8a-fa2d78f47a25" containerName="payload-processing" containerID="cri-o://8a4906584911033305fd2650d829607358fd82f7d2cb94891eaa1babef49cbd0" gracePeriod=25
Nov 03 23:04:03.068160 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:03.068121    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4f9bb0-6f8b69d999-g9rgk" podUID="b9a82175-bff2-4fda-a4ff-756a962b51fe" containerName="payload-pre-processing" containerID="cri-o://a65f72026c07410cd34af55e9af19769a6779dae5ea7725de8cdb10523ff2941" gracePeriod=25
Nov 03 23:04:03.344702 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:03.344674    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4f9bb0-58c4b77cfb-cdgpk"
Nov 03 23:04:03.371886 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:03.371863    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4f9bb0-6f8b69d999-g9rgk"
Nov 03 23:04:03.379267 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:03.379210    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96145989-02fa-41e0-9a8a-fa2d78f47a25-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96145989-02fa-41e0-9a8a-fa2d78f47a25-plugins-config-volume\") pod \"96145989-02fa-41e0-9a8a-fa2d78f47a25\" (UID: \"96145989-02fa-41e0-9a8a-fa2d78f47a25\") "
Nov 03 23:04:03.379398 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:03.379382    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96145989-02fa-41e0-9a8a-fa2d78f47a25-kube-api-access-p7n4h\" (UniqueName: \"kubernetes.io/projected/96145989-02fa-41e0-9a8a-fa2d78f47a25-kube-api-access-p7n4h\") pod \"96145989-02fa-41e0-9a8a-fa2d78f47a25\" (UID: \"96145989-02fa-41e0-9a8a-fa2d78f47a25\") "
Nov 03 23:04:03.379558 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:03.379520    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96145989-02fa-41e0-9a8a-fa2d78f47a25-plugins-config-volume" pod "96145989-02fa-41e0-9a8a-fa2d78f47a25" (UID: "96145989-02fa-41e0-9a8a-fa2d78f47a25"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:04:03.379918 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:03.379893    4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96145989-02fa-41e0-9a8a-fa2d78f47a25-plugins-config-volume\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:04:03.381416 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:03.381396    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96145989-02fa-41e0-9a8a-fa2d78f47a25-kube-api-access-p7n4h" pod "96145989-02fa-41e0-9a8a-fa2d78f47a25" (UID: "96145989-02fa-41e0-9a8a-fa2d78f47a25"). InnerVolumeSpecName "kube-api-access-p7n4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:04:03.480618 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:03.480576    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9a82175-bff2-4fda-a4ff-756a962b51fe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9a82175-bff2-4fda-a4ff-756a962b51fe-plugins-config-volume\") pod \"b9a82175-bff2-4fda-a4ff-756a962b51fe\" (UID: \"b9a82175-bff2-4fda-a4ff-756a962b51fe\") "
Nov 03 23:04:03.480801 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:03.480694    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9a82175-bff2-4fda-a4ff-756a962b51fe-kube-api-access-q7x8f\" (UniqueName: \"kubernetes.io/projected/b9a82175-bff2-4fda-a4ff-756a962b51fe-kube-api-access-q7x8f\") pod \"b9a82175-bff2-4fda-a4ff-756a962b51fe\" (UID: \"b9a82175-bff2-4fda-a4ff-756a962b51fe\") "
Nov 03 23:04:03.480961 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:03.480937    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9a82175-bff2-4fda-a4ff-756a962b51fe-plugins-config-volume" pod "b9a82175-bff2-4fda-a4ff-756a962b51fe" (UID: "b9a82175-bff2-4fda-a4ff-756a962b51fe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:04:03.481078 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:03.481065    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7n4h\" (UniqueName: \"kubernetes.io/projected/96145989-02fa-41e0-9a8a-fa2d78f47a25-kube-api-access-p7n4h\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:04:03.481143 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:03.481079    4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9a82175-bff2-4fda-a4ff-756a962b51fe-plugins-config-volume\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:04:03.482895 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:03.482871    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9a82175-bff2-4fda-a4ff-756a962b51fe-kube-api-access-q7x8f" pod "b9a82175-bff2-4fda-a4ff-756a962b51fe" (UID: "b9a82175-bff2-4fda-a4ff-756a962b51fe"). InnerVolumeSpecName "kube-api-access-q7x8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:04:03.582766 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:03.582685    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7x8f\" (UniqueName: \"kubernetes.io/projected/b9a82175-bff2-4fda-a4ff-756a962b51fe-kube-api-access-q7x8f\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:04:03.757501 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:03.757468    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:04:03.781131 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:03.781093    4431 generic.go:356] "Generic (PLEG): container finished" podID="b9a82175-bff2-4fda-a4ff-756a962b51fe" containerID="a65f72026c07410cd34af55e9af19769a6779dae5ea7725de8cdb10523ff2941" exitCode=0
Nov 03 23:04:03.781300 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:03.781138    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4f9bb0-6f8b69d999-g9rgk"
Nov 03 23:04:03.781300 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:03.781181    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4f9bb0-6f8b69d999-g9rgk" event={"ID":"b9a82175-bff2-4fda-a4ff-756a962b51fe","Type":"ContainerDied","Data":"a65f72026c07410cd34af55e9af19769a6779dae5ea7725de8cdb10523ff2941"}
Nov 03 23:04:03.781300 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:03.781289    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4f9bb0-6f8b69d999-g9rgk" event={"ID":"b9a82175-bff2-4fda-a4ff-756a962b51fe","Type":"ContainerDied","Data":"0eb91ace58723490379bd3cc1d2ce191dfcfa4ed14f2f2b99cae4719531bd449"}
Nov 03 23:04:03.781476 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:03.781337    4431 scope.go:122] "RemoveContainer" containerID="a65f72026c07410cd34af55e9af19769a6779dae5ea7725de8cdb10523ff2941"
Nov 03 23:04:03.782668 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:03.782648    4431 generic.go:356] "Generic (PLEG): container finished" podID="96145989-02fa-41e0-9a8a-fa2d78f47a25" containerID="8a4906584911033305fd2650d829607358fd82f7d2cb94891eaa1babef49cbd0" exitCode=0
Nov 03 23:04:03.782750 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:03.782688    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4f9bb0-58c4b77cfb-cdgpk"
Nov 03 23:04:03.782750 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:03.782730    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4f9bb0-58c4b77cfb-cdgpk" event={"ID":"96145989-02fa-41e0-9a8a-fa2d78f47a25","Type":"ContainerDied","Data":"8a4906584911033305fd2650d829607358fd82f7d2cb94891eaa1babef49cbd0"}
Nov 03 23:04:03.782888 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:03.782783    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4f9bb0-58c4b77cfb-cdgpk" event={"ID":"96145989-02fa-41e0-9a8a-fa2d78f47a25","Type":"ContainerDied","Data":"ae8c5622125d70ff1308e1b3479b5d306fa1a2f2f07158b4b14600c75e10a92a"}
Nov 03 23:04:03.801568 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:03.801543    4431 scope.go:122] "RemoveContainer" containerID="a65f72026c07410cd34af55e9af19769a6779dae5ea7725de8cdb10523ff2941"
Nov 03 23:04:03.801965 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:04:03.801939    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a65f72026c07410cd34af55e9af19769a6779dae5ea7725de8cdb10523ff2941\": container with ID starting with a65f72026c07410cd34af55e9af19769a6779dae5ea7725de8cdb10523ff2941 not found: ID does not exist" containerID="a65f72026c07410cd34af55e9af19769a6779dae5ea7725de8cdb10523ff2941"
Nov 03 23:04:03.802088 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:03.801975    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a65f72026c07410cd34af55e9af19769a6779dae5ea7725de8cdb10523ff2941"} err="failed to get container status \"a65f72026c07410cd34af55e9af19769a6779dae5ea7725de8cdb10523ff2941\": rpc error: code = NotFound desc = could not find container \"a65f72026c07410cd34af55e9af19769a6779dae5ea7725de8cdb10523ff2941\": container with ID starting with a65f72026c07410cd34af55e9af19769a6779dae5ea7725de8cdb10523ff2941 not found: ID does not exist"
Nov 03 23:04:03.802088 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:03.802001    4431 scope.go:122] "RemoveContainer" containerID="8a4906584911033305fd2650d829607358fd82f7d2cb94891eaa1babef49cbd0"
Nov 03 23:04:03.812749 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:03.812727    4431 scope.go:122] "RemoveContainer" containerID="8a4906584911033305fd2650d829607358fd82f7d2cb94891eaa1babef49cbd0"
Nov 03 23:04:03.813068 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:04:03.813048    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a4906584911033305fd2650d829607358fd82f7d2cb94891eaa1babef49cbd0\": container with ID starting with 8a4906584911033305fd2650d829607358fd82f7d2cb94891eaa1babef49cbd0 not found: ID does not exist" containerID="8a4906584911033305fd2650d829607358fd82f7d2cb94891eaa1babef49cbd0"
Nov 03 23:04:03.813134 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:03.813077    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a4906584911033305fd2650d829607358fd82f7d2cb94891eaa1babef49cbd0"} err="failed to get container status \"8a4906584911033305fd2650d829607358fd82f7d2cb94891eaa1babef49cbd0\": rpc error: code = NotFound desc = could not find container \"8a4906584911033305fd2650d829607358fd82f7d2cb94891eaa1babef49cbd0\": container with ID starting with 8a4906584911033305fd2650d829607358fd82f7d2cb94891eaa1babef49cbd0 not found: ID does not exist"
Nov 03 23:04:03.847917 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:03.847849    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4f9bb0-6f8b69d999-g9rgk"]
Nov 03 23:04:03.851931 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:03.851905    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4f9bb0-6f8b69d999-g9rgk"]
Nov 03 23:04:03.892287 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:03.892250    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-4f9bb0-58c4b77cfb-cdgpk"]
Nov 03 23:04:03.895507 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:03.895479    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-4f9bb0-58c4b77cfb-cdgpk"]
Nov 03 23:04:03.899361 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:03.899333    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:04:04.089552 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:04.089520    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96145989-02fa-41e0-9a8a-fa2d78f47a25" path="/var/lib/kubelet/pods/96145989-02fa-41e0-9a8a-fa2d78f47a25/volumes"
Nov 03 23:04:04.089919 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:04.089906    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9a82175-bff2-4fda-a4ff-756a962b51fe" path="/var/lib/kubelet/pods/b9a82175-bff2-4fda-a4ff-756a962b51fe/volumes"
Nov 03 23:04:04.222789 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:04.222754    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:04:05.718639 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:05.718609    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f6dcc0-84b547ffbc-6v2rf"
Nov 03 23:04:05.719033 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:05.719019    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f6dcc0-85d8fff6db-tvftq"
Nov 03 23:04:05.806163 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:05.802387    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:04:07.307460 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:07.307426    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:04:08.301168 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:08.301133    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:04:08.442892 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:08.442856    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:04:08.642570 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:08.642485    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:04:08.922157 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:08.922121    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:04:10.548233 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:10.548177    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:04:12.402877 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:12.402841    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4b994-6f6f95d665-g4gs5"]
Nov 03 23:04:12.403435 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:12.403081    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4b994-6f6f95d665-g4gs5" podUID="309cd370-a7b9-4a9d-84ac-e4e122dec456" containerName="maas-api" containerID="cri-o://ba08c1ded0a276322b565bac9e2134f4bfbaa8840593d52c17fabfc294924224" gracePeriod=30
Nov 03 23:04:12.516594 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:12.516557    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:04:12.675230 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:12.675173    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:04:12.749186 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:12.749156    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4b994-6f6f95d665-g4gs5"
Nov 03 23:04:12.831777 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:12.831740    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:04:12.859092 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:12.859054    4431 generic.go:356] "Generic (PLEG): container finished" podID="309cd370-a7b9-4a9d-84ac-e4e122dec456" containerID="ba08c1ded0a276322b565bac9e2134f4bfbaa8840593d52c17fabfc294924224" exitCode=0
Nov 03 23:04:12.859335 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:12.859101    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4b994-6f6f95d665-g4gs5"
Nov 03 23:04:12.859335 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:12.859119    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4b994-6f6f95d665-g4gs5" event={"ID":"309cd370-a7b9-4a9d-84ac-e4e122dec456","Type":"ContainerDied","Data":"ba08c1ded0a276322b565bac9e2134f4bfbaa8840593d52c17fabfc294924224"}
Nov 03 23:04:12.859335 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:12.859184    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4b994-6f6f95d665-g4gs5" event={"ID":"309cd370-a7b9-4a9d-84ac-e4e122dec456","Type":"ContainerDied","Data":"8a83d76333d9fd12b282b823cb48dc20182fa4859e1031b91dca0acf311cc5f0"}
Nov 03 23:04:12.859335 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:12.859210    4431 scope.go:122] "RemoveContainer" containerID="ba08c1ded0a276322b565bac9e2134f4bfbaa8840593d52c17fabfc294924224"
Nov 03 23:04:12.875871 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:12.875853    4431 scope.go:122] "RemoveContainer" containerID="ba08c1ded0a276322b565bac9e2134f4bfbaa8840593d52c17fabfc294924224"
Nov 03 23:04:12.876261 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:04:12.876200    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba08c1ded0a276322b565bac9e2134f4bfbaa8840593d52c17fabfc294924224\": container with ID starting with ba08c1ded0a276322b565bac9e2134f4bfbaa8840593d52c17fabfc294924224 not found: ID does not exist" containerID="ba08c1ded0a276322b565bac9e2134f4bfbaa8840593d52c17fabfc294924224"
Nov 03 23:04:12.876353 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:12.876267    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba08c1ded0a276322b565bac9e2134f4bfbaa8840593d52c17fabfc294924224"} err="failed to get container status \"ba08c1ded0a276322b565bac9e2134f4bfbaa8840593d52c17fabfc294924224\": rpc error: code = NotFound desc = could not find container \"ba08c1ded0a276322b565bac9e2134f4bfbaa8840593d52c17fabfc294924224\": container with ID starting with ba08c1ded0a276322b565bac9e2134f4bfbaa8840593d52c17fabfc294924224 not found: ID does not exist"
Nov 03 23:04:12.913539 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:12.913503    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/309cd370-a7b9-4a9d-84ac-e4e122dec456-metrics-certs\" (UniqueName: \"kubernetes.io/secret/309cd370-a7b9-4a9d-84ac-e4e122dec456-metrics-certs\") pod \"309cd370-a7b9-4a9d-84ac-e4e122dec456\" (UID: \"309cd370-a7b9-4a9d-84ac-e4e122dec456\") "
Nov 03 23:04:12.913745 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:12.913558    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/309cd370-a7b9-4a9d-84ac-e4e122dec456-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/309cd370-a7b9-4a9d-84ac-e4e122dec456-maas-api-tls\") pod \"309cd370-a7b9-4a9d-84ac-e4e122dec456\" (UID: \"309cd370-a7b9-4a9d-84ac-e4e122dec456\") "
Nov 03 23:04:12.913745 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:12.913730    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/309cd370-a7b9-4a9d-84ac-e4e122dec456-kube-api-access-qhdsp\" (UniqueName: \"kubernetes.io/projected/309cd370-a7b9-4a9d-84ac-e4e122dec456-kube-api-access-qhdsp\") pod \"309cd370-a7b9-4a9d-84ac-e4e122dec456\" (UID: \"309cd370-a7b9-4a9d-84ac-e4e122dec456\") "
Nov 03 23:04:12.913948 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:12.913930    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/309cd370-a7b9-4a9d-84ac-e4e122dec456-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/309cd370-a7b9-4a9d-84ac-e4e122dec456-openshift-service-ca\") pod \"309cd370-a7b9-4a9d-84ac-e4e122dec456\" (UID: \"309cd370-a7b9-4a9d-84ac-e4e122dec456\") "
Nov 03 23:04:12.915585 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:12.915539    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/309cd370-a7b9-4a9d-84ac-e4e122dec456-openshift-service-ca" pod "309cd370-a7b9-4a9d-84ac-e4e122dec456" (UID: "309cd370-a7b9-4a9d-84ac-e4e122dec456"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:04:12.967424 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:12.967374    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/309cd370-a7b9-4a9d-84ac-e4e122dec456-maas-api-tls" pod "309cd370-a7b9-4a9d-84ac-e4e122dec456" (UID: "309cd370-a7b9-4a9d-84ac-e4e122dec456"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:04:12.967623 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:12.967468    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/309cd370-a7b9-4a9d-84ac-e4e122dec456-metrics-certs" pod "309cd370-a7b9-4a9d-84ac-e4e122dec456" (UID: "309cd370-a7b9-4a9d-84ac-e4e122dec456"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:04:12.967623 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:12.967478    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/309cd370-a7b9-4a9d-84ac-e4e122dec456-kube-api-access-qhdsp" pod "309cd370-a7b9-4a9d-84ac-e4e122dec456" (UID: "309cd370-a7b9-4a9d-84ac-e4e122dec456"). InnerVolumeSpecName "kube-api-access-qhdsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:04:13.016935 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:13.016894    4431 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/309cd370-a7b9-4a9d-84ac-e4e122dec456-openshift-service-ca\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:04:13.016935 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:13.016933    4431 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/309cd370-a7b9-4a9d-84ac-e4e122dec456-metrics-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:04:13.017147 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:13.016950    4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/309cd370-a7b9-4a9d-84ac-e4e122dec456-maas-api-tls\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:04:13.017147 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:13.016964    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhdsp\" (UniqueName: \"kubernetes.io/projected/309cd370-a7b9-4a9d-84ac-e4e122dec456-kube-api-access-qhdsp\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:04:13.122383 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:13.122351    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:04:13.206534 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:13.206497    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4b994-6f6f95d665-g4gs5"]
Nov 03 23:04:13.217942 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:13.217869    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4b994-6f6f95d665-g4gs5"]
Nov 03 23:04:13.266452 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:13.266393    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:04:13.750657 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:13.750627    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:04:14.089689 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:14.089614    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="309cd370-a7b9-4a9d-84ac-e4e122dec456" path="/var/lib/kubelet/pods/309cd370-a7b9-4a9d-84ac-e4e122dec456/volumes"
Nov 03 23:04:14.971917 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:14.971884    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:04:17.518120 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:17.518078    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:04:19.108058 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:19.108004    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:04:19.774645 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:19.774613    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:04:19.915148 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:19.915119    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:04:20.124678 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:20.124604    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:04:21.330442 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:21.330414    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:04:23.027253 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:23.027196    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:04:24.206063 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:24.206027    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f9bb0-7444566c76-2zdrh"]
Nov 03 23:04:24.206671 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:24.206294    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f9bb0-7444566c76-2zdrh" podUID="56a3f25c-1413-4374-b87e-223dadff1c76" containerName="maas-api" containerID="cri-o://2cd951f8d941ac6029213de7557d42d3cd8228c4d28e746fe27093783e9bbc29" gracePeriod=30
Nov 03 23:04:24.309446 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:24.309415    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:04:24.317667 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:24.317631    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f9bb0-7444566c76-2zdrh" podUID="56a3f25c-1413-4374-b87e-223dadff1c76" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.143:8443/health\": dial tcp 10.133.0.143:8443: connect: connection refused"
Nov 03 23:04:24.554263 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:24.554224    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f9bb0-7444566c76-2zdrh"
Nov 03 23:04:24.669270 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:24.667743    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56a3f25c-1413-4374-b87e-223dadff1c76-kube-api-access-w6k69\" (UniqueName: \"kubernetes.io/projected/56a3f25c-1413-4374-b87e-223dadff1c76-kube-api-access-w6k69\") pod \"56a3f25c-1413-4374-b87e-223dadff1c76\" (UID: \"56a3f25c-1413-4374-b87e-223dadff1c76\") "
Nov 03 23:04:24.669270 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:24.667804    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56a3f25c-1413-4374-b87e-223dadff1c76-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56a3f25c-1413-4374-b87e-223dadff1c76-maas-api-tls\") pod \"56a3f25c-1413-4374-b87e-223dadff1c76\" (UID: \"56a3f25c-1413-4374-b87e-223dadff1c76\") "
Nov 03 23:04:24.669270 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:24.667869    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56a3f25c-1413-4374-b87e-223dadff1c76-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56a3f25c-1413-4374-b87e-223dadff1c76-openshift-service-ca\") pod \"56a3f25c-1413-4374-b87e-223dadff1c76\" (UID: \"56a3f25c-1413-4374-b87e-223dadff1c76\") "
Nov 03 23:04:24.669270 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:24.667997    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56a3f25c-1413-4374-b87e-223dadff1c76-metrics-certs\" (UniqueName: \"kubernetes.io/secret/56a3f25c-1413-4374-b87e-223dadff1c76-metrics-certs\") pod \"56a3f25c-1413-4374-b87e-223dadff1c76\" (UID: \"56a3f25c-1413-4374-b87e-223dadff1c76\") "
Nov 03 23:04:24.680267 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:24.672632    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56a3f25c-1413-4374-b87e-223dadff1c76-maas-api-tls" pod "56a3f25c-1413-4374-b87e-223dadff1c76" (UID: "56a3f25c-1413-4374-b87e-223dadff1c76"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:04:24.680267 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:24.677157    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56a3f25c-1413-4374-b87e-223dadff1c76-openshift-service-ca" pod "56a3f25c-1413-4374-b87e-223dadff1c76" (UID: "56a3f25c-1413-4374-b87e-223dadff1c76"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:04:24.680611 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:24.680583    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56a3f25c-1413-4374-b87e-223dadff1c76-metrics-certs" pod "56a3f25c-1413-4374-b87e-223dadff1c76" (UID: "56a3f25c-1413-4374-b87e-223dadff1c76"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:04:24.682443 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:24.682418    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56a3f25c-1413-4374-b87e-223dadff1c76-kube-api-access-w6k69" pod "56a3f25c-1413-4374-b87e-223dadff1c76" (UID: "56a3f25c-1413-4374-b87e-223dadff1c76"). InnerVolumeSpecName "kube-api-access-w6k69". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:04:24.769532 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:24.769457    4431 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56a3f25c-1413-4374-b87e-223dadff1c76-openshift-service-ca\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:04:24.769532 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:24.769485    4431 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56a3f25c-1413-4374-b87e-223dadff1c76-metrics-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:04:24.769532 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:24.769496    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6k69\" (UniqueName: \"kubernetes.io/projected/56a3f25c-1413-4374-b87e-223dadff1c76-kube-api-access-w6k69\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:04:24.769532 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:24.769506    4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56a3f25c-1413-4374-b87e-223dadff1c76-maas-api-tls\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:04:24.845476 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:24.845410    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:04:24.938550 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:24.938514    4431 generic.go:356] "Generic (PLEG): container finished" podID="56a3f25c-1413-4374-b87e-223dadff1c76" containerID="2cd951f8d941ac6029213de7557d42d3cd8228c4d28e746fe27093783e9bbc29" exitCode=0
Nov 03 23:04:24.938728 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:24.938562    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f9bb0-7444566c76-2zdrh"
Nov 03 23:04:24.938728 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:24.938581    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f9bb0-7444566c76-2zdrh" event={"ID":"56a3f25c-1413-4374-b87e-223dadff1c76","Type":"ContainerDied","Data":"2cd951f8d941ac6029213de7557d42d3cd8228c4d28e746fe27093783e9bbc29"}
Nov 03 23:04:24.938728 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:24.938637    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f9bb0-7444566c76-2zdrh" event={"ID":"56a3f25c-1413-4374-b87e-223dadff1c76","Type":"ContainerDied","Data":"516c5a66e66ddc8e1e6103913443302999c8bf37a6029e842051a34fca86bad1"}
Nov 03 23:04:24.938728 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:24.938666    4431 scope.go:122] "RemoveContainer" containerID="2cd951f8d941ac6029213de7557d42d3cd8228c4d28e746fe27093783e9bbc29"
Nov 03 23:04:24.949044 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:24.949025    4431 scope.go:122] "RemoveContainer" containerID="2cd951f8d941ac6029213de7557d42d3cd8228c4d28e746fe27093783e9bbc29"
Nov 03 23:04:24.949370 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:04:24.949344    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cd951f8d941ac6029213de7557d42d3cd8228c4d28e746fe27093783e9bbc29\": container with ID starting with 2cd951f8d941ac6029213de7557d42d3cd8228c4d28e746fe27093783e9bbc29 not found: ID does not exist" containerID="2cd951f8d941ac6029213de7557d42d3cd8228c4d28e746fe27093783e9bbc29"
Nov 03 23:04:24.949440 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:24.949380    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cd951f8d941ac6029213de7557d42d3cd8228c4d28e746fe27093783e9bbc29"} err="failed to get container status \"2cd951f8d941ac6029213de7557d42d3cd8228c4d28e746fe27093783e9bbc29\": rpc error: code = NotFound desc = could not find container \"2cd951f8d941ac6029213de7557d42d3cd8228c4d28e746fe27093783e9bbc29\": container with ID starting with 2cd951f8d941ac6029213de7557d42d3cd8228c4d28e746fe27093783e9bbc29 not found: ID does not exist"
Nov 03 23:04:24.990995 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:24.990969    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f9bb0-7444566c76-2zdrh"]
Nov 03 23:04:25.002980 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:25.002945    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4f9bb0-7444566c76-2zdrh"]
Nov 03 23:04:25.007033 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:25.007004    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:04:25.157603 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:25.157490    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:04:25.488826 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:25.488795    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:04:26.088667 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:26.088635    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56a3f25c-1413-4374-b87e-223dadff1c76" path="/var/lib/kubelet/pods/56a3f25c-1413-4374-b87e-223dadff1c76/volumes"
Nov 03 23:04:26.777276 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:26.777231    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:04:26.922532 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:26.922488    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:04:27.134171 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:27.134093    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:04:28.414747 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:28.414711    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:04:28.855788 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:28.855707    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f"]
Nov 03 23:04:28.856369 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:28.856351    4431 cpu_manager.go:414] "containerMap: removing container" podUID="56a3f25c-1413-4374-b87e-223dadff1c76" containerName="maas-api"
Nov 03 23:04:28.856452 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:28.856372    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56a3f25c-1413-4374-b87e-223dadff1c76" containerName="maas-api"
Nov 03 23:04:28.856452 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:28.856394    4431 cpu_manager.go:414] "containerMap: removing container" podUID="309cd370-a7b9-4a9d-84ac-e4e122dec456" containerName="maas-api"
Nov 03 23:04:28.856452 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:28.856399    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="309cd370-a7b9-4a9d-84ac-e4e122dec456" containerName="maas-api"
Nov 03 23:04:28.856452 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:28.856428    4431 cpu_manager.go:414] "containerMap: removing container" podUID="96145989-02fa-41e0-9a8a-fa2d78f47a25" containerName="payload-processing"
Nov 03 23:04:28.856452 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:28.856434    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96145989-02fa-41e0-9a8a-fa2d78f47a25" containerName="payload-processing"
Nov 03 23:04:28.856452 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:28.856441    4431 cpu_manager.go:414] "containerMap: removing container" podUID="b9a82175-bff2-4fda-a4ff-756a962b51fe" containerName="payload-pre-processing"
Nov 03 23:04:28.856452 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:28.856446    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9a82175-bff2-4fda-a4ff-756a962b51fe" containerName="payload-pre-processing"
Nov 03 23:04:28.856728 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:28.856531    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9a82175-bff2-4fda-a4ff-756a962b51fe" containerName="payload-pre-processing"
Nov 03 23:04:28.856728 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:28.856544    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="56a3f25c-1413-4374-b87e-223dadff1c76" containerName="maas-api"
Nov 03 23:04:28.856728 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:28.856552    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="309cd370-a7b9-4a9d-84ac-e4e122dec456" containerName="maas-api"
Nov 03 23:04:28.856728 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:28.856563    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="96145989-02fa-41e0-9a8a-fa2d78f47a25" containerName="payload-processing"
Nov 03 23:04:28.861715 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:28.861694    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f"
Nov 03 23:04:28.896938 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:28.896901    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f"]
Nov 03 23:04:28.926426 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:28.926394    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a04d44db-0f1c-49c5-aa5b-069a256aa783-istio-podinfo\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f\" (UID: \"a04d44db-0f1c-49c5-aa5b-069a256aa783\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f"
Nov 03 23:04:28.926603 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:28.926465    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a04d44db-0f1c-49c5-aa5b-069a256aa783-workload-socket\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f\" (UID: \"a04d44db-0f1c-49c5-aa5b-069a256aa783\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f"
Nov 03 23:04:28.926603 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:28.926521    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a04d44db-0f1c-49c5-aa5b-069a256aa783-credential-socket\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f\" (UID: \"a04d44db-0f1c-49c5-aa5b-069a256aa783\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f"
Nov 03 23:04:28.926686 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:28.926644    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a04d44db-0f1c-49c5-aa5b-069a256aa783-istio-token\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f\" (UID: \"a04d44db-0f1c-49c5-aa5b-069a256aa783\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f"
Nov 03 23:04:28.926726 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:28.926701    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss9vx\" (UniqueName: \"kubernetes.io/projected/a04d44db-0f1c-49c5-aa5b-069a256aa783-kube-api-access-ss9vx\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f\" (UID: \"a04d44db-0f1c-49c5-aa5b-069a256aa783\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f"
Nov 03 23:04:28.926766 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:28.926734    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a04d44db-0f1c-49c5-aa5b-069a256aa783-workload-certs\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f\" (UID: \"a04d44db-0f1c-49c5-aa5b-069a256aa783\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f"
Nov 03 23:04:28.926820 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:28.926795    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a04d44db-0f1c-49c5-aa5b-069a256aa783-istio-envoy\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f\" (UID: \"a04d44db-0f1c-49c5-aa5b-069a256aa783\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f"
Nov 03 23:04:28.926960 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:28.926942    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a04d44db-0f1c-49c5-aa5b-069a256aa783-istiod-ca-cert\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f\" (UID: \"a04d44db-0f1c-49c5-aa5b-069a256aa783\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f"
Nov 03 23:04:28.927062 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:28.927047    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a04d44db-0f1c-49c5-aa5b-069a256aa783-istio-data\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f\" (UID: \"a04d44db-0f1c-49c5-aa5b-069a256aa783\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f"
Nov 03 23:04:29.028699 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:29.028667    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a04d44db-0f1c-49c5-aa5b-069a256aa783-istio-podinfo\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f\" (UID: \"a04d44db-0f1c-49c5-aa5b-069a256aa783\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f"
Nov 03 23:04:29.028866 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:29.028715    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a04d44db-0f1c-49c5-aa5b-069a256aa783-workload-socket\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f\" (UID: \"a04d44db-0f1c-49c5-aa5b-069a256aa783\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f"
Nov 03 23:04:29.028866 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:29.028752    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a04d44db-0f1c-49c5-aa5b-069a256aa783-credential-socket\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f\" (UID: \"a04d44db-0f1c-49c5-aa5b-069a256aa783\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f"
Nov 03 23:04:29.028866 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:29.028795    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a04d44db-0f1c-49c5-aa5b-069a256aa783-istio-token\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f\" (UID: \"a04d44db-0f1c-49c5-aa5b-069a256aa783\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f"
Nov 03 23:04:29.028866 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:29.028828    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss9vx\" (UniqueName: \"kubernetes.io/projected/a04d44db-0f1c-49c5-aa5b-069a256aa783-kube-api-access-ss9vx\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f\" (UID: \"a04d44db-0f1c-49c5-aa5b-069a256aa783\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f"
Nov 03 23:04:29.028866 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:29.028858    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a04d44db-0f1c-49c5-aa5b-069a256aa783-workload-certs\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f\" (UID: \"a04d44db-0f1c-49c5-aa5b-069a256aa783\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f"
Nov 03 23:04:29.029186 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:29.028908    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a04d44db-0f1c-49c5-aa5b-069a256aa783-istio-envoy\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f\" (UID: \"a04d44db-0f1c-49c5-aa5b-069a256aa783\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f"
Nov 03 23:04:29.029186 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:29.028976    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a04d44db-0f1c-49c5-aa5b-069a256aa783-istiod-ca-cert\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f\" (UID: \"a04d44db-0f1c-49c5-aa5b-069a256aa783\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f"
Nov 03 23:04:29.029186 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:29.029060    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a04d44db-0f1c-49c5-aa5b-069a256aa783-istio-data\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f\" (UID: \"a04d44db-0f1c-49c5-aa5b-069a256aa783\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f"
Nov 03 23:04:29.029186 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:29.029156    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a04d44db-0f1c-49c5-aa5b-069a256aa783-workload-socket\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f\" (UID: \"a04d44db-0f1c-49c5-aa5b-069a256aa783\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f"
Nov 03 23:04:29.029449 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:29.029250    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a04d44db-0f1c-49c5-aa5b-069a256aa783-credential-socket\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f\" (UID: \"a04d44db-0f1c-49c5-aa5b-069a256aa783\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f"
Nov 03 23:04:29.029449 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:29.029323    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a04d44db-0f1c-49c5-aa5b-069a256aa783-workload-certs\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f\" (UID: \"a04d44db-0f1c-49c5-aa5b-069a256aa783\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f"
Nov 03 23:04:29.029449 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:29.029402    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a04d44db-0f1c-49c5-aa5b-069a256aa783-istio-data\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f\" (UID: \"a04d44db-0f1c-49c5-aa5b-069a256aa783\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f"
Nov 03 23:04:29.029860 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:29.029837    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a04d44db-0f1c-49c5-aa5b-069a256aa783-istiod-ca-cert\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f\" (UID: \"a04d44db-0f1c-49c5-aa5b-069a256aa783\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f"
Nov 03 23:04:29.031159 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:29.031139    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a04d44db-0f1c-49c5-aa5b-069a256aa783-istio-envoy\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f\" (UID: \"a04d44db-0f1c-49c5-aa5b-069a256aa783\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f"
Nov 03 23:04:29.031450 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:29.031435    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a04d44db-0f1c-49c5-aa5b-069a256aa783-istio-podinfo\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f\" (UID: \"a04d44db-0f1c-49c5-aa5b-069a256aa783\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f"
Nov 03 23:04:29.051455 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:29.051424    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a04d44db-0f1c-49c5-aa5b-069a256aa783-istio-token\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f\" (UID: \"a04d44db-0f1c-49c5-aa5b-069a256aa783\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f"
Nov 03 23:04:29.051720 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:29.051697    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss9vx\" (UniqueName: \"kubernetes.io/projected/a04d44db-0f1c-49c5-aa5b-069a256aa783-kube-api-access-ss9vx\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f\" (UID: \"a04d44db-0f1c-49c5-aa5b-069a256aa783\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f"
Nov 03 23:04:29.180021 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:29.179991    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f"
Nov 03 23:04:29.499105 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:29.499081    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f"]
Nov 03 23:04:29.500950 ip-10-0-139-73 kubenswrapper[4431]: W1103 23:04:29.500923    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda04d44db_0f1c_49c5_aa5b_069a256aa783.slice/crio-0edd7333a66e1481f4305ec168085f6b68fe2b62efdeb4b5115db0f6d48ede41 WatchSource:0}: Error finding container 0edd7333a66e1481f4305ec168085f6b68fe2b62efdeb4b5115db0f6d48ede41: Status 404 returned error can't find the container with id 0edd7333a66e1481f4305ec168085f6b68fe2b62efdeb4b5115db0f6d48ede41
Nov 03 23:04:29.503758 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:29.503720    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:04:29.503889 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:29.503808    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:04:29.503889 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:29.503854    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:04:29.980086 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:29.979974    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f" event={"ID":"a04d44db-0f1c-49c5-aa5b-069a256aa783","Type":"ContainerStarted","Data":"f427517d211dcaf69950307e501de1630bbba1fa5221ce4cc180f1ec7e063803"}
Nov 03 23:04:29.980086 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:29.980046    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f" event={"ID":"a04d44db-0f1c-49c5-aa5b-069a256aa783","Type":"ContainerStarted","Data":"0edd7333a66e1481f4305ec168085f6b68fe2b62efdeb4b5115db0f6d48ede41"}
Nov 03 23:04:30.027484 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:30.027426    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f" podStartSLOduration=2.027410525 podStartE2EDuration="2.027410525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:04:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:04:30.02487444 +0000 UTC m=+3094.050730303" watchObservedRunningTime="2026-11-03 23:04:30.027410525 +0000 UTC m=+3094.053266387"
Nov 03 23:04:30.126942 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:30.126903    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:04:30.184610 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:30.184580    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f"
Nov 03 23:04:30.186515 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:30.186485    4431 patch_prober.go:29] interesting pod/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body=
Nov 03 23:04:30.186775 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:30.186738    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f" podUID="a04d44db-0f1c-49c5-aa5b-069a256aa783" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused"
Nov 03 23:04:31.180339 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:31.180302    4431 patch_prober.go:29] interesting pod/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body=
Nov 03 23:04:31.180774 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:31.180357    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f" podUID="a04d44db-0f1c-49c5-aa5b-069a256aa783" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused"
Nov 03 23:04:31.284995 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:04:31.284963    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 23:04:31.784579 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:31.784542    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:04:31.928551 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:31.928504    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:04:32.132823 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:32.132731    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t"]
Nov 03 23:04:32.133015 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:32.132980    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t" podUID="fe16bcab-2fb0-425d-961c-adc59648bace" containerName="istio-proxy" containerID="cri-o://dfda9e3c3722843e060da24db1ea7b25c40cae65892de7873e46747a41991127" gracePeriod=30
Nov 03 23:04:32.180631 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:32.180587    4431 patch_prober.go:29] interesting pod/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body=
Nov 03 23:04:32.181199 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:32.181167    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f" podUID="a04d44db-0f1c-49c5-aa5b-069a256aa783" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused"
Nov 03 23:04:32.200350 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:32.200322    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:04:32.664859 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:32.664818    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:04:33.181350 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:33.181319    4431 patch_prober.go:29] interesting pod/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body=
Nov 03 23:04:33.181749 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:33.181371    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f" podUID="a04d44db-0f1c-49c5-aa5b-069a256aa783" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused"
Nov 03 23:04:33.565126 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:33.565033    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:04:34.181063 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:34.181024    4431 patch_prober.go:29] interesting pod/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body=
Nov 03 23:04:34.181327 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:34.181115    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f" podUID="a04d44db-0f1c-49c5-aa5b-069a256aa783" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused"
Nov 03 23:04:34.882707 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:34.882668    4431 patch_prober.go:29] interesting pod/e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 23:04:34.883110 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:34.882730    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t" podUID="fe16bcab-2fb0-425d-961c-adc59648bace" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 23:04:35.181112 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:35.181079    4431 patch_prober.go:29] interesting pod/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body=
Nov 03 23:04:35.181367 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:35.181130    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f" podUID="a04d44db-0f1c-49c5-aa5b-069a256aa783" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused"
Nov 03 23:04:35.272157 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:35.272126    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:04:36.184913 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:36.184877    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f"
Nov 03 23:04:36.185433 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:36.185409    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f"
Nov 03 23:04:36.186156 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:36.186134    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f"
Nov 03 23:04:36.889286 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:36.889254    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:04:37.398801 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:37.398742    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t"
Nov 03 23:04:37.560945 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:37.560905    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe16bcab-2fb0-425d-961c-adc59648bace-istio-token\" (UniqueName: \"kubernetes.io/projected/fe16bcab-2fb0-425d-961c-adc59648bace-istio-token\") pod \"fe16bcab-2fb0-425d-961c-adc59648bace\" (UID: \"fe16bcab-2fb0-425d-961c-adc59648bace\") "
Nov 03 23:04:37.561153 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:37.560993    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe16bcab-2fb0-425d-961c-adc59648bace-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe16bcab-2fb0-425d-961c-adc59648bace-workload-socket\") pod \"fe16bcab-2fb0-425d-961c-adc59648bace\" (UID: \"fe16bcab-2fb0-425d-961c-adc59648bace\") "
Nov 03 23:04:37.561153 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:37.561064    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe16bcab-2fb0-425d-961c-adc59648bace-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe16bcab-2fb0-425d-961c-adc59648bace-istiod-ca-cert\") pod \"fe16bcab-2fb0-425d-961c-adc59648bace\" (UID: \"fe16bcab-2fb0-425d-961c-adc59648bace\") "
Nov 03 23:04:37.561153 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:37.561096    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fe16bcab-2fb0-425d-961c-adc59648bace-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe16bcab-2fb0-425d-961c-adc59648bace-istio-podinfo\") pod \"fe16bcab-2fb0-425d-961c-adc59648bace\" (UID: \"fe16bcab-2fb0-425d-961c-adc59648bace\") "
Nov 03 23:04:37.561405 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:37.561181    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe16bcab-2fb0-425d-961c-adc59648bace-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe16bcab-2fb0-425d-961c-adc59648bace-workload-certs\") pod \"fe16bcab-2fb0-425d-961c-adc59648bace\" (UID: \"fe16bcab-2fb0-425d-961c-adc59648bace\") "
Nov 03 23:04:37.561405 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:37.561277    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe16bcab-2fb0-425d-961c-adc59648bace-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe16bcab-2fb0-425d-961c-adc59648bace-istio-envoy\") pod \"fe16bcab-2fb0-425d-961c-adc59648bace\" (UID: \"fe16bcab-2fb0-425d-961c-adc59648bace\") "
Nov 03 23:04:37.561405 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:37.561336    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe16bcab-2fb0-425d-961c-adc59648bace-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe16bcab-2fb0-425d-961c-adc59648bace-credential-socket\") pod \"fe16bcab-2fb0-425d-961c-adc59648bace\" (UID: \"fe16bcab-2fb0-425d-961c-adc59648bace\") "
Nov 03 23:04:37.561571 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:37.561435    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe16bcab-2fb0-425d-961c-adc59648bace-kube-api-access-8bs4n\" (UniqueName: \"kubernetes.io/projected/fe16bcab-2fb0-425d-961c-adc59648bace-kube-api-access-8bs4n\") pod \"fe16bcab-2fb0-425d-961c-adc59648bace\" (UID: \"fe16bcab-2fb0-425d-961c-adc59648bace\") "
Nov 03 23:04:37.561571 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:37.561469    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe16bcab-2fb0-425d-961c-adc59648bace-workload-certs" pod "fe16bcab-2fb0-425d-961c-adc59648bace" (UID: "fe16bcab-2fb0-425d-961c-adc59648bace"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:04:37.561571 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:37.561491    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe16bcab-2fb0-425d-961c-adc59648bace-workload-socket" pod "fe16bcab-2fb0-425d-961c-adc59648bace" (UID: "fe16bcab-2fb0-425d-961c-adc59648bace"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:04:37.561571 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:37.561523    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe16bcab-2fb0-425d-961c-adc59648bace-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe16bcab-2fb0-425d-961c-adc59648bace-istio-data\") pod \"fe16bcab-2fb0-425d-961c-adc59648bace\" (UID: \"fe16bcab-2fb0-425d-961c-adc59648bace\") "
Nov 03 23:04:37.561780 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:37.561677    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe16bcab-2fb0-425d-961c-adc59648bace-credential-socket" pod "fe16bcab-2fb0-425d-961c-adc59648bace" (UID: "fe16bcab-2fb0-425d-961c-adc59648bace"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:04:37.561935 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:37.561884    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe16bcab-2fb0-425d-961c-adc59648bace-istio-data" pod "fe16bcab-2fb0-425d-961c-adc59648bace" (UID: "fe16bcab-2fb0-425d-961c-adc59648bace"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:04:37.562048 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:37.562001    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe16bcab-2fb0-425d-961c-adc59648bace-istiod-ca-cert" pod "fe16bcab-2fb0-425d-961c-adc59648bace" (UID: "fe16bcab-2fb0-425d-961c-adc59648bace"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:04:37.562164 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:37.562148    4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe16bcab-2fb0-425d-961c-adc59648bace-workload-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:04:37.562263 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:37.562168    4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe16bcab-2fb0-425d-961c-adc59648bace-istiod-ca-cert\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:04:37.562263 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:37.562184    4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe16bcab-2fb0-425d-961c-adc59648bace-workload-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:04:37.562263 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:37.562196    4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe16bcab-2fb0-425d-961c-adc59648bace-credential-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:04:37.562263 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:37.562210    4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe16bcab-2fb0-425d-961c-adc59648bace-istio-data\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:04:37.563750 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:37.563725    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe16bcab-2fb0-425d-961c-adc59648bace-istio-token" pod "fe16bcab-2fb0-425d-961c-adc59648bace" (UID: "fe16bcab-2fb0-425d-961c-adc59648bace"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:04:37.563869 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:37.563820    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fe16bcab-2fb0-425d-961c-adc59648bace-istio-podinfo" pod "fe16bcab-2fb0-425d-961c-adc59648bace" (UID: "fe16bcab-2fb0-425d-961c-adc59648bace"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 23:04:37.563933 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:37.563919    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe16bcab-2fb0-425d-961c-adc59648bace-kube-api-access-8bs4n" pod "fe16bcab-2fb0-425d-961c-adc59648bace" (UID: "fe16bcab-2fb0-425d-961c-adc59648bace"). InnerVolumeSpecName "kube-api-access-8bs4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:04:37.564056 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:37.564025    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe16bcab-2fb0-425d-961c-adc59648bace-istio-envoy" pod "fe16bcab-2fb0-425d-961c-adc59648bace" (UID: "fe16bcab-2fb0-425d-961c-adc59648bace"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:04:37.579791 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:37.579759    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-553753/distinct-2-w2-553753-kserve-5bf8568844-7q4l2"]
Nov 03 23:04:37.599024 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:37.598991    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-553753/simulator-w2-553753-kserve-64d77d4dfc-4rlbn"]
Nov 03 23:04:37.663427 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:37.663399    4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe16bcab-2fb0-425d-961c-adc59648bace-istio-podinfo\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:04:37.663427 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:37.663425    4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe16bcab-2fb0-425d-961c-adc59648bace-istio-envoy\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:04:37.663633 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:37.663434    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bs4n\" (UniqueName: \"kubernetes.io/projected/fe16bcab-2fb0-425d-961c-adc59648bace-kube-api-access-8bs4n\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:04:37.663633 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:37.663443    4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe16bcab-2fb0-425d-961c-adc59648bace-istio-token\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:04:37.736647 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:37.736612    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:04:38.034874 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:38.034798    4431 generic.go:356] "Generic (PLEG): container finished" podID="fe16bcab-2fb0-425d-961c-adc59648bace" containerID="dfda9e3c3722843e060da24db1ea7b25c40cae65892de7873e46747a41991127" exitCode=0
Nov 03 23:04:38.034874 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:38.034851    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t"
Nov 03 23:04:38.035062 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:38.034878    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t" event={"ID":"fe16bcab-2fb0-425d-961c-adc59648bace","Type":"ContainerDied","Data":"dfda9e3c3722843e060da24db1ea7b25c40cae65892de7873e46747a41991127"}
Nov 03 23:04:38.035062 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:38.034966    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t" event={"ID":"fe16bcab-2fb0-425d-961c-adc59648bace","Type":"ContainerDied","Data":"92202e432711cef9fb152fc9acbdc700d8cf6d1924e852577b9e815d8e5005e5"}
Nov 03 23:04:38.035062 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:38.034996    4431 scope.go:122] "RemoveContainer" containerID="dfda9e3c3722843e060da24db1ea7b25c40cae65892de7873e46747a41991127"
Nov 03 23:04:38.046160 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:38.046140    4431 scope.go:122] "RemoveContainer" containerID="e0814691e68fe0fa1842c60813e59af9d1da0e745e0a14e3b15866d7a228c8c1"
Nov 03 23:04:38.055528 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:38.055510    4431 scope.go:122] "RemoveContainer" containerID="dfda9e3c3722843e060da24db1ea7b25c40cae65892de7873e46747a41991127"
Nov 03 23:04:38.055786 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:04:38.055766    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfda9e3c3722843e060da24db1ea7b25c40cae65892de7873e46747a41991127\": container with ID starting with dfda9e3c3722843e060da24db1ea7b25c40cae65892de7873e46747a41991127 not found: ID does not exist" containerID="dfda9e3c3722843e060da24db1ea7b25c40cae65892de7873e46747a41991127"
Nov 03 23:04:38.055834 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:38.055795    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfda9e3c3722843e060da24db1ea7b25c40cae65892de7873e46747a41991127"} err="failed to get container status \"dfda9e3c3722843e060da24db1ea7b25c40cae65892de7873e46747a41991127\": rpc error: code = NotFound desc = could not find container \"dfda9e3c3722843e060da24db1ea7b25c40cae65892de7873e46747a41991127\": container with ID starting with dfda9e3c3722843e060da24db1ea7b25c40cae65892de7873e46747a41991127 not found: ID does not exist"
Nov 03 23:04:38.055834 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:38.055811    4431 scope.go:122] "RemoveContainer" containerID="e0814691e68fe0fa1842c60813e59af9d1da0e745e0a14e3b15866d7a228c8c1"
Nov 03 23:04:38.056042 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:04:38.056021    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0814691e68fe0fa1842c60813e59af9d1da0e745e0a14e3b15866d7a228c8c1\": container with ID starting with e0814691e68fe0fa1842c60813e59af9d1da0e745e0a14e3b15866d7a228c8c1 not found: ID does not exist" containerID="e0814691e68fe0fa1842c60813e59af9d1da0e745e0a14e3b15866d7a228c8c1"
Nov 03 23:04:38.056084 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:38.056048    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0814691e68fe0fa1842c60813e59af9d1da0e745e0a14e3b15866d7a228c8c1"} err="failed to get container status \"e0814691e68fe0fa1842c60813e59af9d1da0e745e0a14e3b15866d7a228c8c1\": rpc error: code = NotFound desc = could not find container \"e0814691e68fe0fa1842c60813e59af9d1da0e745e0a14e3b15866d7a228c8c1\": container with ID starting with e0814691e68fe0fa1842c60813e59af9d1da0e745e0a14e3b15866d7a228c8c1 not found: ID does not exist"
Nov 03 23:04:38.089916 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:38.089891    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t"]
Nov 03 23:04:38.100440 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:38.100411    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-4f9bb0-openshift-default-599dcdf4f9-pwn4t"]
Nov 03 23:04:38.570041 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:38.569990    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:04:40.056903 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:40.056866    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:04:40.090757 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:40.090723    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe16bcab-2fb0-425d-961c-adc59648bace" path="/var/lib/kubelet/pods/fe16bcab-2fb0-425d-961c-adc59648bace/volumes"
Nov 03 23:04:41.554469 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:41.554436    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:04:43.118252 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:43.118206    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:04:43.412162 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:43.412114    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:04:43.548542 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:43.548499    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:04:44.734780 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:44.734743    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:04:46.318257 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:46.318200    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:04:47.917129 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:47.917081    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:04:49.916284 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:49.916247    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:04:50.614883 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:50.614852    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:04:51.262309 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:51.262273    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:04:51.428188 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:51.428157    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:04:51.569870 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:51.569784    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:04:51.636656 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:51.636624    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-162d7b-6c77f794d7-6cqjh"]
Nov 03 23:04:51.636867 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:51.636840    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-162d7b-6c77f794d7-6cqjh" podUID="0d83968f-285d-4804-a015-7bddf4586765" containerName="maas-api" containerID="cri-o://f8abe050bdc6c3d99b320e93888b59702ed9332bbe1432ea52f56526cb75b2a5" gracePeriod=30
Nov 03 23:04:51.731455 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:51.731416    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:04:51.811239 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:51.810860    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-162d7b-699f7476d4-tdvfk"]
Nov 03 23:04:51.813335 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:51.811206    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-162d7b-699f7476d4-tdvfk" podUID="61d2223e-d3b7-42ec-956d-ac7b3f94180c" containerName="payload-processing" containerID="cri-o://cc1980887317183449b97da7603ad579393d8e518a3c549586fdc49a697ce009" gracePeriod=30
Nov 03 23:04:51.839298 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:51.838882    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-162d7b-74ddd95c5-rpng8"]
Nov 03 23:04:51.839697 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:51.839387    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-162d7b-74ddd95c5-rpng8" podUID="72eb7f3b-8510-4699-8764-72c129b39943" containerName="payload-pre-processing" containerID="cri-o://92b2c77138f684a8e9a52c48b23c5b1ae00c2521b4ddf0e79b3bc01c12aed7e9" gracePeriod=30
Nov 03 23:04:51.901280 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:51.899945    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:04:52.049260 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:52.047049    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:04:52.062272 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:52.062130    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-162d7b-6c77f794d7-6cqjh"
Nov 03 23:04:52.182241 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:52.167753    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d83968f-285d-4804-a015-7bddf4586765-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d83968f-285d-4804-a015-7bddf4586765-maas-api-tls\") pod \"0d83968f-285d-4804-a015-7bddf4586765\" (UID: \"0d83968f-285d-4804-a015-7bddf4586765\") "
Nov 03 23:04:52.182241 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:52.167844    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d83968f-285d-4804-a015-7bddf4586765-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d83968f-285d-4804-a015-7bddf4586765-metrics-certs\") pod \"0d83968f-285d-4804-a015-7bddf4586765\" (UID: \"0d83968f-285d-4804-a015-7bddf4586765\") "
Nov 03 23:04:52.182241 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:52.167892    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d83968f-285d-4804-a015-7bddf4586765-kube-api-access-cw8mn\" (UniqueName: \"kubernetes.io/projected/0d83968f-285d-4804-a015-7bddf4586765-kube-api-access-cw8mn\") pod \"0d83968f-285d-4804-a015-7bddf4586765\" (UID: \"0d83968f-285d-4804-a015-7bddf4586765\") "
Nov 03 23:04:52.182241 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:52.167933    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d83968f-285d-4804-a015-7bddf4586765-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d83968f-285d-4804-a015-7bddf4586765-openshift-service-ca\") pod \"0d83968f-285d-4804-a015-7bddf4586765\" (UID: \"0d83968f-285d-4804-a015-7bddf4586765\") "
Nov 03 23:04:52.182241 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:52.169672    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d83968f-285d-4804-a015-7bddf4586765-openshift-service-ca" pod "0d83968f-285d-4804-a015-7bddf4586765" (UID: "0d83968f-285d-4804-a015-7bddf4586765"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:04:52.186169 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:52.186039    4431 generic.go:356] "Generic (PLEG): container finished" podID="72eb7f3b-8510-4699-8764-72c129b39943" containerID="92b2c77138f684a8e9a52c48b23c5b1ae00c2521b4ddf0e79b3bc01c12aed7e9" exitCode=0
Nov 03 23:04:52.207611 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:52.186579    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-162d7b-74ddd95c5-rpng8" event={"ID":"72eb7f3b-8510-4699-8764-72c129b39943","Type":"ContainerDied","Data":"92b2c77138f684a8e9a52c48b23c5b1ae00c2521b4ddf0e79b3bc01c12aed7e9"}
Nov 03 23:04:52.229864 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:52.228088    4431 generic.go:356] "Generic (PLEG): container finished" podID="61d2223e-d3b7-42ec-956d-ac7b3f94180c" containerID="cc1980887317183449b97da7603ad579393d8e518a3c549586fdc49a697ce009" exitCode=0
Nov 03 23:04:52.229864 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:52.228157    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-162d7b-699f7476d4-tdvfk" event={"ID":"61d2223e-d3b7-42ec-956d-ac7b3f94180c","Type":"ContainerDied","Data":"cc1980887317183449b97da7603ad579393d8e518a3c549586fdc49a697ce009"}
Nov 03 23:04:52.231737 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:52.230955    4431 generic.go:356] "Generic (PLEG): container finished" podID="0d83968f-285d-4804-a015-7bddf4586765" containerID="f8abe050bdc6c3d99b320e93888b59702ed9332bbe1432ea52f56526cb75b2a5" exitCode=0
Nov 03 23:04:52.231737 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:52.230997    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-162d7b-6c77f794d7-6cqjh" event={"ID":"0d83968f-285d-4804-a015-7bddf4586765","Type":"ContainerDied","Data":"f8abe050bdc6c3d99b320e93888b59702ed9332bbe1432ea52f56526cb75b2a5"}
Nov 03 23:04:52.231737 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:52.231041    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-162d7b-6c77f794d7-6cqjh" event={"ID":"0d83968f-285d-4804-a015-7bddf4586765","Type":"ContainerDied","Data":"64b0e5377517596575b7507c87ae83f3a1685e4afeec63dccb495249e00cb0ed"}
Nov 03 23:04:52.231737 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:52.231071    4431 scope.go:122] "RemoveContainer" containerID="f8abe050bdc6c3d99b320e93888b59702ed9332bbe1432ea52f56526cb75b2a5"
Nov 03 23:04:52.231737 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:52.231244    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-162d7b-6c77f794d7-6cqjh"
Nov 03 23:04:52.233451 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:52.233420    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d83968f-285d-4804-a015-7bddf4586765-maas-api-tls" pod "0d83968f-285d-4804-a015-7bddf4586765" (UID: "0d83968f-285d-4804-a015-7bddf4586765"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:04:52.233576 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:52.233534    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d83968f-285d-4804-a015-7bddf4586765-metrics-certs" pod "0d83968f-285d-4804-a015-7bddf4586765" (UID: "0d83968f-285d-4804-a015-7bddf4586765"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:04:52.237252 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:52.234388    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-162d7b-699f7476d4-tdvfk"
Nov 03 23:04:52.242482 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:52.242424    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d83968f-285d-4804-a015-7bddf4586765-kube-api-access-cw8mn" pod "0d83968f-285d-4804-a015-7bddf4586765" (UID: "0d83968f-285d-4804-a015-7bddf4586765"). InnerVolumeSpecName "kube-api-access-cw8mn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:04:52.254501 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:52.254482    4431 scope.go:122] "RemoveContainer" containerID="f8abe050bdc6c3d99b320e93888b59702ed9332bbe1432ea52f56526cb75b2a5"
Nov 03 23:04:52.254921 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:04:52.254889    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8abe050bdc6c3d99b320e93888b59702ed9332bbe1432ea52f56526cb75b2a5\": container with ID starting with f8abe050bdc6c3d99b320e93888b59702ed9332bbe1432ea52f56526cb75b2a5 not found: ID does not exist" containerID="f8abe050bdc6c3d99b320e93888b59702ed9332bbe1432ea52f56526cb75b2a5"
Nov 03 23:04:52.255047 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:52.255029    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8abe050bdc6c3d99b320e93888b59702ed9332bbe1432ea52f56526cb75b2a5"} err="failed to get container status \"f8abe050bdc6c3d99b320e93888b59702ed9332bbe1432ea52f56526cb75b2a5\": rpc error: code = NotFound desc = could not find container \"f8abe050bdc6c3d99b320e93888b59702ed9332bbe1432ea52f56526cb75b2a5\": container with ID starting with f8abe050bdc6c3d99b320e93888b59702ed9332bbe1432ea52f56526cb75b2a5 not found: ID does not exist"
Nov 03 23:04:52.269812 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:52.269781    4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d83968f-285d-4804-a015-7bddf4586765-maas-api-tls\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:04:52.269812 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:52.269810    4431 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d83968f-285d-4804-a015-7bddf4586765-metrics-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:04:52.270361 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:52.269820    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cw8mn\" (UniqueName: \"kubernetes.io/projected/0d83968f-285d-4804-a015-7bddf4586765-kube-api-access-cw8mn\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:04:52.270361 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:52.269829    4431 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d83968f-285d-4804-a015-7bddf4586765-openshift-service-ca\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:04:52.294400 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:52.294374    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-162d7b-74ddd95c5-rpng8"
Nov 03 23:04:52.370750 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:52.370666    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61d2223e-d3b7-42ec-956d-ac7b3f94180c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61d2223e-d3b7-42ec-956d-ac7b3f94180c-plugins-config-volume\") pod \"61d2223e-d3b7-42ec-956d-ac7b3f94180c\" (UID: \"61d2223e-d3b7-42ec-956d-ac7b3f94180c\") "
Nov 03 23:04:52.370750 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:52.370708    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72eb7f3b-8510-4699-8764-72c129b39943-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72eb7f3b-8510-4699-8764-72c129b39943-plugins-config-volume\") pod \"72eb7f3b-8510-4699-8764-72c129b39943\" (UID: \"72eb7f3b-8510-4699-8764-72c129b39943\") "
Nov 03 23:04:52.371011 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:52.370875    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61d2223e-d3b7-42ec-956d-ac7b3f94180c-kube-api-access-xj7rs\" (UniqueName: \"kubernetes.io/projected/61d2223e-d3b7-42ec-956d-ac7b3f94180c-kube-api-access-xj7rs\") pod \"61d2223e-d3b7-42ec-956d-ac7b3f94180c\" (UID: \"61d2223e-d3b7-42ec-956d-ac7b3f94180c\") "
Nov 03 23:04:52.371011 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:52.370982    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72eb7f3b-8510-4699-8764-72c129b39943-kube-api-access-7j85h\" (UniqueName: \"kubernetes.io/projected/72eb7f3b-8510-4699-8764-72c129b39943-kube-api-access-7j85h\") pod \"72eb7f3b-8510-4699-8764-72c129b39943\" (UID: \"72eb7f3b-8510-4699-8764-72c129b39943\") "
Nov 03 23:04:52.371174 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:52.371144    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61d2223e-d3b7-42ec-956d-ac7b3f94180c-plugins-config-volume" pod "61d2223e-d3b7-42ec-956d-ac7b3f94180c" (UID: "61d2223e-d3b7-42ec-956d-ac7b3f94180c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:04:52.371261 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:52.371195    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72eb7f3b-8510-4699-8764-72c129b39943-plugins-config-volume" pod "72eb7f3b-8510-4699-8764-72c129b39943" (UID: "72eb7f3b-8510-4699-8764-72c129b39943"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:04:52.371648 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:52.371600    4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61d2223e-d3b7-42ec-956d-ac7b3f94180c-plugins-config-volume\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:04:52.371775 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:52.371651    4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72eb7f3b-8510-4699-8764-72c129b39943-plugins-config-volume\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:04:52.373116 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:52.373094    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72eb7f3b-8510-4699-8764-72c129b39943-kube-api-access-7j85h" pod "72eb7f3b-8510-4699-8764-72c129b39943" (UID: "72eb7f3b-8510-4699-8764-72c129b39943"). InnerVolumeSpecName "kube-api-access-7j85h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:04:52.373207 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:52.373137    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61d2223e-d3b7-42ec-956d-ac7b3f94180c-kube-api-access-xj7rs" pod "61d2223e-d3b7-42ec-956d-ac7b3f94180c" (UID: "61d2223e-d3b7-42ec-956d-ac7b3f94180c"). InnerVolumeSpecName "kube-api-access-xj7rs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:04:52.473233 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:52.473185    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xj7rs\" (UniqueName: \"kubernetes.io/projected/61d2223e-d3b7-42ec-956d-ac7b3f94180c-kube-api-access-xj7rs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:04:52.473233 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:52.473227    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7j85h\" (UniqueName: \"kubernetes.io/projected/72eb7f3b-8510-4699-8764-72c129b39943-kube-api-access-7j85h\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:04:52.542144 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:52.542114    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:04:52.579640 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:52.579588    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-162d7b-6c77f794d7-6cqjh"]
Nov 03 23:04:52.586855 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:52.586819    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-162d7b-6c77f794d7-6cqjh"]
Nov 03 23:04:52.601690 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:52.601646    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-553753/distinct-2-w2-553753-kserve-5bf8568844-7q4l2" podUID="a1894b80-164c-4ce1-94f3-1e5011d9759e" containerName="main" containerID="cri-o://bf6a6fbd1cd9ef11331329778a45ce7bb8ed898bc7ed45b815d4a45bcd5118e5" gracePeriod=45
Nov 03 23:04:52.623849 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:52.623785    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-553753/simulator-w2-553753-kserve-64d77d4dfc-4rlbn" podUID="a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e" containerName="main" containerID="cri-o://4b94a3ff05734cad306a301f260f406dd2548c0943f2b6b894ce910563dbbb81" gracePeriod=45
Nov 03 23:04:52.694047 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:52.694014    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:04:52.921945 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:52.921922    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-553753/distinct-2-w2-553753-kserve-5bf8568844-7q4l2"
Nov 03 23:04:52.925896 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:52.925874    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-553753/simulator-w2-553753-kserve-64d77d4dfc-4rlbn"
Nov 03 23:04:52.981752 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:52.981716    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e-tmp-dir\") pod \"a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e\" (UID: \"a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e\") "
Nov 03 23:04:52.981946 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:52.981815    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e-model-cache\") pod \"a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e\" (UID: \"a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e\") "
Nov 03 23:04:52.981946 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:52.981908    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e-home\" (UniqueName: \"kubernetes.io/empty-dir/a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e-home\") pod \"a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e\" (UID: \"a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e\") "
Nov 03 23:04:52.982203 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:52.982155    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e-tmp-dir" pod "a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e" (UID: "a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:04:52.982352 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:52.982334    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e-home" pod "a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e" (UID: "a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:04:52.982439 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:52.982388    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e-model-cache" pod "a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e" (UID: "a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:04:52.982500 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:52.982459    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1894b80-164c-4ce1-94f3-1e5011d9759e-kube-api-access-kwxwf\" (UniqueName: \"kubernetes.io/projected/a1894b80-164c-4ce1-94f3-1e5011d9759e-kube-api-access-kwxwf\") pod \"a1894b80-164c-4ce1-94f3-1e5011d9759e\" (UID: \"a1894b80-164c-4ce1-94f3-1e5011d9759e\") "
Nov 03 23:04:52.982814 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:52.982490    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1894b80-164c-4ce1-94f3-1e5011d9759e-tls-certs\" (UniqueName: \"kubernetes.io/secret/a1894b80-164c-4ce1-94f3-1e5011d9759e-tls-certs\") pod \"a1894b80-164c-4ce1-94f3-1e5011d9759e\" (UID: \"a1894b80-164c-4ce1-94f3-1e5011d9759e\") "
Nov 03 23:04:52.982906 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:52.982863    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1894b80-164c-4ce1-94f3-1e5011d9759e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1894b80-164c-4ce1-94f3-1e5011d9759e-tmp-dir\") pod \"a1894b80-164c-4ce1-94f3-1e5011d9759e\" (UID: \"a1894b80-164c-4ce1-94f3-1e5011d9759e\") "
Nov 03 23:04:52.983165 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:52.983136    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1894b80-164c-4ce1-94f3-1e5011d9759e-tmp-dir" pod "a1894b80-164c-4ce1-94f3-1e5011d9759e" (UID: "a1894b80-164c-4ce1-94f3-1e5011d9759e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:04:52.983758 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:52.983489    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1894b80-164c-4ce1-94f3-1e5011d9759e-home\" (UniqueName: \"kubernetes.io/empty-dir/a1894b80-164c-4ce1-94f3-1e5011d9759e-home\") pod \"a1894b80-164c-4ce1-94f3-1e5011d9759e\" (UID: \"a1894b80-164c-4ce1-94f3-1e5011d9759e\") "
Nov 03 23:04:52.984095 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:52.983741    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1894b80-164c-4ce1-94f3-1e5011d9759e-home" pod "a1894b80-164c-4ce1-94f3-1e5011d9759e" (UID: "a1894b80-164c-4ce1-94f3-1e5011d9759e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:04:52.985481 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:52.984260    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e-tls-certs\" (UniqueName: \"kubernetes.io/secret/a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e-tls-certs\") pod \"a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e\" (UID: \"a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e\") "
Nov 03 23:04:52.985481 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:52.984304    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e-dshm\") pod \"a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e\" (UID: \"a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e\") "
Nov 03 23:04:52.985481 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:52.984381    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1894b80-164c-4ce1-94f3-1e5011d9759e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1894b80-164c-4ce1-94f3-1e5011d9759e-dshm\") pod \"a1894b80-164c-4ce1-94f3-1e5011d9759e\" (UID: \"a1894b80-164c-4ce1-94f3-1e5011d9759e\") "
Nov 03 23:04:52.985481 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:52.984409    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1894b80-164c-4ce1-94f3-1e5011d9759e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1894b80-164c-4ce1-94f3-1e5011d9759e-model-cache\") pod \"a1894b80-164c-4ce1-94f3-1e5011d9759e\" (UID: \"a1894b80-164c-4ce1-94f3-1e5011d9759e\") "
Nov 03 23:04:52.985481 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:52.984462    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e-kube-api-access-tflqt\" (UniqueName: \"kubernetes.io/projected/a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e-kube-api-access-tflqt\") pod \"a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e\" (UID: \"a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e\") "
Nov 03 23:04:52.985481 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:52.985009    4431 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e-tmp-dir\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:04:52.985481 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:52.985028    4431 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e-model-cache\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:04:52.985481 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:52.985041    4431 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e-home\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:04:52.985481 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:52.985054    4431 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1894b80-164c-4ce1-94f3-1e5011d9759e-tmp-dir\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:04:52.985481 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:52.985065    4431 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1894b80-164c-4ce1-94f3-1e5011d9759e-home\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:04:52.986322 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:52.985671    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1894b80-164c-4ce1-94f3-1e5011d9759e-model-cache" pod "a1894b80-164c-4ce1-94f3-1e5011d9759e" (UID: "a1894b80-164c-4ce1-94f3-1e5011d9759e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:04:52.986533 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:52.986500    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1894b80-164c-4ce1-94f3-1e5011d9759e-tls-certs" pod "a1894b80-164c-4ce1-94f3-1e5011d9759e" (UID: "a1894b80-164c-4ce1-94f3-1e5011d9759e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:04:52.988659 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:52.988631    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e-tls-certs" pod "a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e" (UID: "a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:04:52.989378 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:52.989342    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e-kube-api-access-tflqt" pod "a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e" (UID: "a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e"). InnerVolumeSpecName "kube-api-access-tflqt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:04:52.989504 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:52.989410    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e-dshm" pod "a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e" (UID: "a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:04:52.989743 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:52.989711    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1894b80-164c-4ce1-94f3-1e5011d9759e-dshm" pod "a1894b80-164c-4ce1-94f3-1e5011d9759e" (UID: "a1894b80-164c-4ce1-94f3-1e5011d9759e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:04:52.993393 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:52.993368    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1894b80-164c-4ce1-94f3-1e5011d9759e-kube-api-access-kwxwf" pod "a1894b80-164c-4ce1-94f3-1e5011d9759e" (UID: "a1894b80-164c-4ce1-94f3-1e5011d9759e"). InnerVolumeSpecName "kube-api-access-kwxwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:04:53.086407 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:53.086378    4431 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1894b80-164c-4ce1-94f3-1e5011d9759e-dshm\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:04:53.086407 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:53.086406    4431 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1894b80-164c-4ce1-94f3-1e5011d9759e-model-cache\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:04:53.086619 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:53.086417    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tflqt\" (UniqueName: \"kubernetes.io/projected/a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e-kube-api-access-tflqt\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:04:53.086619 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:53.086426    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwxwf\" (UniqueName: \"kubernetes.io/projected/a1894b80-164c-4ce1-94f3-1e5011d9759e-kube-api-access-kwxwf\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:04:53.086619 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:53.086435    4431 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1894b80-164c-4ce1-94f3-1e5011d9759e-tls-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:04:53.086619 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:53.086442    4431 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e-tls-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:04:53.086619 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:53.086450    4431 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e-dshm\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:04:53.226921 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:53.226882    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:04:53.253252 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:53.252427    4431 generic.go:356] "Generic (PLEG): container finished" podID="a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e" containerID="4b94a3ff05734cad306a301f260f406dd2548c0943f2b6b894ce910563dbbb81" exitCode=0
Nov 03 23:04:53.253252 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:53.252573    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-553753/simulator-w2-553753-kserve-64d77d4dfc-4rlbn" event={"ID":"a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e","Type":"ContainerDied","Data":"4b94a3ff05734cad306a301f260f406dd2548c0943f2b6b894ce910563dbbb81"}
Nov 03 23:04:53.253252 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:53.252611    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-553753/simulator-w2-553753-kserve-64d77d4dfc-4rlbn" event={"ID":"a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e","Type":"ContainerDied","Data":"2f278fb9858b58015c0b65491f3c3d2c6c443c23ee776ca971b60f4b557e4c7f"}
Nov 03 23:04:53.253252 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:53.252635    4431 scope.go:122] "RemoveContainer" containerID="4b94a3ff05734cad306a301f260f406dd2548c0943f2b6b894ce910563dbbb81"
Nov 03 23:04:53.253252 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:53.252732    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-553753/simulator-w2-553753-kserve-64d77d4dfc-4rlbn"
Nov 03 23:04:53.259106 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:53.259074    4431 generic.go:356] "Generic (PLEG): container finished" podID="a1894b80-164c-4ce1-94f3-1e5011d9759e" containerID="bf6a6fbd1cd9ef11331329778a45ce7bb8ed898bc7ed45b815d4a45bcd5118e5" exitCode=0
Nov 03 23:04:53.260050 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:53.260016    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-553753/distinct-2-w2-553753-kserve-5bf8568844-7q4l2" event={"ID":"a1894b80-164c-4ce1-94f3-1e5011d9759e","Type":"ContainerDied","Data":"bf6a6fbd1cd9ef11331329778a45ce7bb8ed898bc7ed45b815d4a45bcd5118e5"}
Nov 03 23:04:53.260242 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:53.260203    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-553753/distinct-2-w2-553753-kserve-5bf8568844-7q4l2" event={"ID":"a1894b80-164c-4ce1-94f3-1e5011d9759e","Type":"ContainerDied","Data":"9f2a9116560e76b9273d350f5a500cb7f7c5df22ffe572bacfd1be7fcd4eaf50"}
Nov 03 23:04:53.260424 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:53.260409    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-553753/distinct-2-w2-553753-kserve-5bf8568844-7q4l2"
Nov 03 23:04:53.301014 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:53.300981    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-162d7b-699f7476d4-tdvfk" event={"ID":"61d2223e-d3b7-42ec-956d-ac7b3f94180c","Type":"ContainerDied","Data":"ed511801f19085657cfc239dcc93f25c0e8f18c85e28501f7e6f8bd539c2d55f"}
Nov 03 23:04:53.301475 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:53.301100    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-162d7b-699f7476d4-tdvfk"
Nov 03 23:04:53.304386 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:53.304351    4431 scope.go:122] "RemoveContainer" containerID="4b94a3ff05734cad306a301f260f406dd2548c0943f2b6b894ce910563dbbb81"
Nov 03 23:04:53.304979 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:04:53.304926    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b94a3ff05734cad306a301f260f406dd2548c0943f2b6b894ce910563dbbb81\": container with ID starting with 4b94a3ff05734cad306a301f260f406dd2548c0943f2b6b894ce910563dbbb81 not found: ID does not exist" containerID="4b94a3ff05734cad306a301f260f406dd2548c0943f2b6b894ce910563dbbb81"
Nov 03 23:04:53.304979 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:53.304965    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b94a3ff05734cad306a301f260f406dd2548c0943f2b6b894ce910563dbbb81"} err="failed to get container status \"4b94a3ff05734cad306a301f260f406dd2548c0943f2b6b894ce910563dbbb81\": rpc error: code = NotFound desc = could not find container \"4b94a3ff05734cad306a301f260f406dd2548c0943f2b6b894ce910563dbbb81\": container with ID starting with 4b94a3ff05734cad306a301f260f406dd2548c0943f2b6b894ce910563dbbb81 not found: ID does not exist"
Nov 03 23:04:53.305205 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:53.304991    4431 scope.go:122] "RemoveContainer" containerID="bf6a6fbd1cd9ef11331329778a45ce7bb8ed898bc7ed45b815d4a45bcd5118e5"
Nov 03 23:04:53.307026 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:53.307003    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-162d7b-74ddd95c5-rpng8" event={"ID":"72eb7f3b-8510-4699-8764-72c129b39943","Type":"ContainerDied","Data":"1d0cb7c2af4eb39ef0d691acf67fec49db0a2ce371ad8853b4d954fa9855a653"}
Nov 03 23:04:53.307138 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:53.307102    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-162d7b-74ddd95c5-rpng8"
Nov 03 23:04:53.317729 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:53.317711    4431 scope.go:122] "RemoveContainer" containerID="bf6a6fbd1cd9ef11331329778a45ce7bb8ed898bc7ed45b815d4a45bcd5118e5"
Nov 03 23:04:53.318001 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:04:53.317981    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf6a6fbd1cd9ef11331329778a45ce7bb8ed898bc7ed45b815d4a45bcd5118e5\": container with ID starting with bf6a6fbd1cd9ef11331329778a45ce7bb8ed898bc7ed45b815d4a45bcd5118e5 not found: ID does not exist" containerID="bf6a6fbd1cd9ef11331329778a45ce7bb8ed898bc7ed45b815d4a45bcd5118e5"
Nov 03 23:04:53.318062 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:53.318009    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf6a6fbd1cd9ef11331329778a45ce7bb8ed898bc7ed45b815d4a45bcd5118e5"} err="failed to get container status \"bf6a6fbd1cd9ef11331329778a45ce7bb8ed898bc7ed45b815d4a45bcd5118e5\": rpc error: code = NotFound desc = could not find container \"bf6a6fbd1cd9ef11331329778a45ce7bb8ed898bc7ed45b815d4a45bcd5118e5\": container with ID starting with bf6a6fbd1cd9ef11331329778a45ce7bb8ed898bc7ed45b815d4a45bcd5118e5 not found: ID does not exist"
Nov 03 23:04:53.318062 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:53.318026    4431 scope.go:122] "RemoveContainer" containerID="cc1980887317183449b97da7603ad579393d8e518a3c549586fdc49a697ce009"
Nov 03 23:04:53.330702 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:53.330682    4431 scope.go:122] "RemoveContainer" containerID="92b2c77138f684a8e9a52c48b23c5b1ae00c2521b4ddf0e79b3bc01c12aed7e9"
Nov 03 23:04:53.354139 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:53.354108    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-553753/simulator-w2-553753-kserve-64d77d4dfc-4rlbn"]
Nov 03 23:04:53.360171 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:53.360146    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-553753/simulator-w2-553753-kserve-64d77d4dfc-4rlbn"]
Nov 03 23:04:53.406486 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:53.406456    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-553753/distinct-2-w2-553753-kserve-5bf8568844-7q4l2"]
Nov 03 23:04:53.412863 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:53.412833    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-553753/distinct-2-w2-553753-kserve-5bf8568844-7q4l2"]
Nov 03 23:04:53.445713 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:53.445660    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-162d7b-699f7476d4-tdvfk"]
Nov 03 23:04:53.454686 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:53.454652    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-162d7b-699f7476d4-tdvfk"]
Nov 03 23:04:53.484442 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:53.484323    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-162d7b-74ddd95c5-rpng8"]
Nov 03 23:04:53.487367 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:53.487339    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-162d7b-74ddd95c5-rpng8"]
Nov 03 23:04:54.089106 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:54.089076    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d83968f-285d-4804-a015-7bddf4586765" path="/var/lib/kubelet/pods/0d83968f-285d-4804-a015-7bddf4586765/volumes"
Nov 03 23:04:54.089550 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:54.089535    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61d2223e-d3b7-42ec-956d-ac7b3f94180c" path="/var/lib/kubelet/pods/61d2223e-d3b7-42ec-956d-ac7b3f94180c/volumes"
Nov 03 23:04:54.089901 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:54.089889    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72eb7f3b-8510-4699-8764-72c129b39943" path="/var/lib/kubelet/pods/72eb7f3b-8510-4699-8764-72c129b39943/volumes"
Nov 03 23:04:54.090589 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:54.090575    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1894b80-164c-4ce1-94f3-1e5011d9759e" path="/var/lib/kubelet/pods/a1894b80-164c-4ce1-94f3-1e5011d9759e/volumes"
Nov 03 23:04:54.090977 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:54.090965    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e" path="/var/lib/kubelet/pods/a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e/volumes"
Nov 03 23:04:54.333531 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:54.333495    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:04:55.655311 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:55.655273    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:04:55.999319 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:55.999286    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:04:57.695444 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:57.695410    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:04:58.991629 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:58.991596    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:04:59.154153 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:59.154113    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:04:59.442238 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:59.441521    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:04:59.609323 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:04:59.609239    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:00.234300 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:00.234260    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:00.374870 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:00.374832    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:00.577932 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:00.577840    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:01.261408 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:05:01.261372    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 23:05:01.311038 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:01.310993    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:02.720349 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:02.720304    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:03.002013 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:03.001928    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:04.338202 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:04.338168    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f6dcc0-85d8fff6db-tvftq"]
Nov 03 23:05:04.370702 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:04.370668    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f6dcc0-84b547ffbc-6v2rf"]
Nov 03 23:05:04.512572 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:04.512525    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:04.647929 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:04.647836    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:06.187067 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:06.187030    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:07.726439 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:07.726403    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:09.339330 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:09.339289    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-f6dcc0-85d8fff6db-tvftq" podUID="9b7dd4de-ba44-4b67-a140-55d8008f5c6f" containerName="payload-processing" containerID="cri-o://d556d32c2863bdbe5bc5c9c768764ec31152937133c520c17a780bc7850edcc1" gracePeriod=25
Nov 03 23:05:09.371653 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:09.371613    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f6dcc0-84b547ffbc-6v2rf" podUID="8dea78d3-8945-4582-8fdb-38b0a28ad416" containerName="payload-pre-processing" containerID="cri-o://83617f2d69af0616a4cc35ea4634b168c338ae955dabfb655387bfce9cfc4343" gracePeriod=25
Nov 03 23:05:09.473781 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:09.473751    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:09.642369 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:09.642345    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f6dcc0-85d8fff6db-tvftq"
Nov 03 23:05:09.675865 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:09.675839    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f6dcc0-84b547ffbc-6v2rf"
Nov 03 23:05:09.708772 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:09.708737    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b7dd4de-ba44-4b67-a140-55d8008f5c6f-kube-api-access-ws724\" (UniqueName: \"kubernetes.io/projected/9b7dd4de-ba44-4b67-a140-55d8008f5c6f-kube-api-access-ws724\") pod \"9b7dd4de-ba44-4b67-a140-55d8008f5c6f\" (UID: \"9b7dd4de-ba44-4b67-a140-55d8008f5c6f\") "
Nov 03 23:05:09.709571 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:09.709537    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b7dd4de-ba44-4b67-a140-55d8008f5c6f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b7dd4de-ba44-4b67-a140-55d8008f5c6f-plugins-config-volume\") pod \"9b7dd4de-ba44-4b67-a140-55d8008f5c6f\" (UID: \"9b7dd4de-ba44-4b67-a140-55d8008f5c6f\") "
Nov 03 23:05:09.710264 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:09.710070    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b7dd4de-ba44-4b67-a140-55d8008f5c6f-plugins-config-volume" pod "9b7dd4de-ba44-4b67-a140-55d8008f5c6f" (UID: "9b7dd4de-ba44-4b67-a140-55d8008f5c6f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:05:09.710944 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:09.710921    4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b7dd4de-ba44-4b67-a140-55d8008f5c6f-plugins-config-volume\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:05:09.711720 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:09.711692    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b7dd4de-ba44-4b67-a140-55d8008f5c6f-kube-api-access-ws724" pod "9b7dd4de-ba44-4b67-a140-55d8008f5c6f" (UID: "9b7dd4de-ba44-4b67-a140-55d8008f5c6f"). InnerVolumeSpecName "kube-api-access-ws724". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:05:09.812178 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:09.812147    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dea78d3-8945-4582-8fdb-38b0a28ad416-kube-api-access-mclqt\" (UniqueName: \"kubernetes.io/projected/8dea78d3-8945-4582-8fdb-38b0a28ad416-kube-api-access-mclqt\") pod \"8dea78d3-8945-4582-8fdb-38b0a28ad416\" (UID: \"8dea78d3-8945-4582-8fdb-38b0a28ad416\") "
Nov 03 23:05:09.812385 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:09.812314    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8dea78d3-8945-4582-8fdb-38b0a28ad416-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dea78d3-8945-4582-8fdb-38b0a28ad416-plugins-config-volume\") pod \"8dea78d3-8945-4582-8fdb-38b0a28ad416\" (UID: \"8dea78d3-8945-4582-8fdb-38b0a28ad416\") "
Nov 03 23:05:09.812676 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:09.812652    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dea78d3-8945-4582-8fdb-38b0a28ad416-plugins-config-volume" pod "8dea78d3-8945-4582-8fdb-38b0a28ad416" (UID: "8dea78d3-8945-4582-8fdb-38b0a28ad416"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:05:09.812808 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:09.812771    4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dea78d3-8945-4582-8fdb-38b0a28ad416-plugins-config-volume\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:05:09.812808 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:09.812785    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ws724\" (UniqueName: \"kubernetes.io/projected/9b7dd4de-ba44-4b67-a140-55d8008f5c6f-kube-api-access-ws724\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:05:09.814256 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:09.814237    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dea78d3-8945-4582-8fdb-38b0a28ad416-kube-api-access-mclqt" pod "8dea78d3-8945-4582-8fdb-38b0a28ad416" (UID: "8dea78d3-8945-4582-8fdb-38b0a28ad416"). InnerVolumeSpecName "kube-api-access-mclqt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:05:09.914574 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:09.914540    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mclqt\" (UniqueName: \"kubernetes.io/projected/8dea78d3-8945-4582-8fdb-38b0a28ad416-kube-api-access-mclqt\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:05:10.430737 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:10.430699    4431 generic.go:356] "Generic (PLEG): container finished" podID="8dea78d3-8945-4582-8fdb-38b0a28ad416" containerID="83617f2d69af0616a4cc35ea4634b168c338ae955dabfb655387bfce9cfc4343" exitCode=0
Nov 03 23:05:10.431191 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:10.430746    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f6dcc0-84b547ffbc-6v2rf"
Nov 03 23:05:10.431191 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:10.430777    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f6dcc0-84b547ffbc-6v2rf" event={"ID":"8dea78d3-8945-4582-8fdb-38b0a28ad416","Type":"ContainerDied","Data":"83617f2d69af0616a4cc35ea4634b168c338ae955dabfb655387bfce9cfc4343"}
Nov 03 23:05:10.431191 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:10.430833    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f6dcc0-84b547ffbc-6v2rf" event={"ID":"8dea78d3-8945-4582-8fdb-38b0a28ad416","Type":"ContainerDied","Data":"24252ec472fe4c53f9213534e99127843e2819dc6a86902c4fb887abc1a4d795"}
Nov 03 23:05:10.431191 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:10.430855    4431 scope.go:122] "RemoveContainer" containerID="83617f2d69af0616a4cc35ea4634b168c338ae955dabfb655387bfce9cfc4343"
Nov 03 23:05:10.432327 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:10.432297    4431 generic.go:356] "Generic (PLEG): container finished" podID="9b7dd4de-ba44-4b67-a140-55d8008f5c6f" containerID="d556d32c2863bdbe5bc5c9c768764ec31152937133c520c17a780bc7850edcc1" exitCode=0
Nov 03 23:05:10.432456 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:10.432326    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f6dcc0-85d8fff6db-tvftq" event={"ID":"9b7dd4de-ba44-4b67-a140-55d8008f5c6f","Type":"ContainerDied","Data":"d556d32c2863bdbe5bc5c9c768764ec31152937133c520c17a780bc7850edcc1"}
Nov 03 23:05:10.432456 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:10.432394    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f6dcc0-85d8fff6db-tvftq" event={"ID":"9b7dd4de-ba44-4b67-a140-55d8008f5c6f","Type":"ContainerDied","Data":"911190bf90088e0869b0fc88c449adcfb07cd9b2c4f476250adb028bdd526013"}
Nov 03 23:05:10.432456 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:10.432343    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f6dcc0-85d8fff6db-tvftq"
Nov 03 23:05:10.440976 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:10.440954    4431 scope.go:122] "RemoveContainer" containerID="83617f2d69af0616a4cc35ea4634b168c338ae955dabfb655387bfce9cfc4343"
Nov 03 23:05:10.441262 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:05:10.441245    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83617f2d69af0616a4cc35ea4634b168c338ae955dabfb655387bfce9cfc4343\": container with ID starting with 83617f2d69af0616a4cc35ea4634b168c338ae955dabfb655387bfce9cfc4343 not found: ID does not exist" containerID="83617f2d69af0616a4cc35ea4634b168c338ae955dabfb655387bfce9cfc4343"
Nov 03 23:05:10.441384 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:10.441269    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83617f2d69af0616a4cc35ea4634b168c338ae955dabfb655387bfce9cfc4343"} err="failed to get container status \"83617f2d69af0616a4cc35ea4634b168c338ae955dabfb655387bfce9cfc4343\": rpc error: code = NotFound desc = could not find container \"83617f2d69af0616a4cc35ea4634b168c338ae955dabfb655387bfce9cfc4343\": container with ID starting with 83617f2d69af0616a4cc35ea4634b168c338ae955dabfb655387bfce9cfc4343 not found: ID does not exist"
Nov 03 23:05:10.441384 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:10.441287    4431 scope.go:122] "RemoveContainer" containerID="d556d32c2863bdbe5bc5c9c768764ec31152937133c520c17a780bc7850edcc1"
Nov 03 23:05:10.450253 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:10.450230    4431 scope.go:122] "RemoveContainer" containerID="d556d32c2863bdbe5bc5c9c768764ec31152937133c520c17a780bc7850edcc1"
Nov 03 23:05:10.450521 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:05:10.450503    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d556d32c2863bdbe5bc5c9c768764ec31152937133c520c17a780bc7850edcc1\": container with ID starting with d556d32c2863bdbe5bc5c9c768764ec31152937133c520c17a780bc7850edcc1 not found: ID does not exist" containerID="d556d32c2863bdbe5bc5c9c768764ec31152937133c520c17a780bc7850edcc1"
Nov 03 23:05:10.450580 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:10.450528    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d556d32c2863bdbe5bc5c9c768764ec31152937133c520c17a780bc7850edcc1"} err="failed to get container status \"d556d32c2863bdbe5bc5c9c768764ec31152937133c520c17a780bc7850edcc1\": rpc error: code = NotFound desc = could not find container \"d556d32c2863bdbe5bc5c9c768764ec31152937133c520c17a780bc7850edcc1\": container with ID starting with d556d32c2863bdbe5bc5c9c768764ec31152937133c520c17a780bc7850edcc1 not found: ID does not exist"
Nov 03 23:05:10.482537 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:10.482511    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f6dcc0-85d8fff6db-tvftq"]
Nov 03 23:05:10.494306 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:10.494283    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f6dcc0-85d8fff6db-tvftq"]
Nov 03 23:05:10.527834 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:10.527801    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f6dcc0-84b547ffbc-6v2rf"]
Nov 03 23:05:10.542533 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:10.536594    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f6dcc0-84b547ffbc-6v2rf"]
Nov 03 23:05:10.697723 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:10.697690    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:10.835027 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:10.834991    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:11.017277 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:11.017151    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:11.315573 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:11.315479    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-b023a2-f5bd44854-vjwb2"]
Nov 03 23:05:11.371699 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:11.371665    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-b023a2-678fcf5d79-x2g2r"]
Nov 03 23:05:11.513318 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:11.513285    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:11.673781 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:11.673749    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:12.089323 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:12.089247    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8dea78d3-8945-4582-8fdb-38b0a28ad416" path="/var/lib/kubelet/pods/8dea78d3-8945-4582-8fdb-38b0a28ad416/volumes"
Nov 03 23:05:12.089656 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:12.089643    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b7dd4de-ba44-4b67-a140-55d8008f5c6f" path="/var/lib/kubelet/pods/9b7dd4de-ba44-4b67-a140-55d8008f5c6f/volumes"
Nov 03 23:05:12.606991 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:12.606952    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:14.138791 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:14.138759    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:15.134295 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:15.134258    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:15.273352 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:15.273318    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:15.749061 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:15.749023    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:15.893227 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:15.893185    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:16.316135 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:16.316077    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-b023a2-f5bd44854-vjwb2" podUID="444d998c-f1d3-46d1-8a77-89eded48361b" containerName="payload-processing" containerID="cri-o://40694285abbdc4f03139caa5c0632d85944095a3cd816759f7dbf875a5648ccf" gracePeriod=25
Nov 03 23:05:16.372921 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:16.372876    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-b023a2-678fcf5d79-x2g2r" podUID="2ae1c3c5-f16b-4e41-b70a-64999f5ab09b" containerName="payload-pre-processing" containerID="cri-o://8740867a7cdc96212e7da8ea9e87c29dd2eea39b6f7119c7e11a579798da967b" gracePeriod=25
Nov 03 23:05:16.474516 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:16.474484    4431 generic.go:356] "Generic (PLEG): container finished" podID="444d998c-f1d3-46d1-8a77-89eded48361b" containerID="40694285abbdc4f03139caa5c0632d85944095a3cd816759f7dbf875a5648ccf" exitCode=0
Nov 03 23:05:16.474681 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:16.474546    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-b023a2-f5bd44854-vjwb2" event={"ID":"444d998c-f1d3-46d1-8a77-89eded48361b","Type":"ContainerDied","Data":"40694285abbdc4f03139caa5c0632d85944095a3cd816759f7dbf875a5648ccf"}
Nov 03 23:05:16.690436 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:16.690402    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-b023a2-f5bd44854-vjwb2"
Nov 03 23:05:16.743954 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:16.743924    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-b023a2-678fcf5d79-x2g2r"
Nov 03 23:05:16.807259 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:16.807210    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ae1c3c5-f16b-4e41-b70a-64999f5ab09b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ae1c3c5-f16b-4e41-b70a-64999f5ab09b-plugins-config-volume\") pod \"2ae1c3c5-f16b-4e41-b70a-64999f5ab09b\" (UID: \"2ae1c3c5-f16b-4e41-b70a-64999f5ab09b\") "
Nov 03 23:05:16.807486 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:16.807331    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/444d998c-f1d3-46d1-8a77-89eded48361b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/444d998c-f1d3-46d1-8a77-89eded48361b-plugins-config-volume\") pod \"444d998c-f1d3-46d1-8a77-89eded48361b\" (UID: \"444d998c-f1d3-46d1-8a77-89eded48361b\") "
Nov 03 23:05:16.807486 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:16.807456    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/444d998c-f1d3-46d1-8a77-89eded48361b-kube-api-access-g9bfw\" (UniqueName: \"kubernetes.io/projected/444d998c-f1d3-46d1-8a77-89eded48361b-kube-api-access-g9bfw\") pod \"444d998c-f1d3-46d1-8a77-89eded48361b\" (UID: \"444d998c-f1d3-46d1-8a77-89eded48361b\") "
Nov 03 23:05:16.807686 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:16.807657    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ae1c3c5-f16b-4e41-b70a-64999f5ab09b-plugins-config-volume" pod "2ae1c3c5-f16b-4e41-b70a-64999f5ab09b" (UID: "2ae1c3c5-f16b-4e41-b70a-64999f5ab09b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:05:16.807919 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:16.807865    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/444d998c-f1d3-46d1-8a77-89eded48361b-plugins-config-volume" pod "444d998c-f1d3-46d1-8a77-89eded48361b" (UID: "444d998c-f1d3-46d1-8a77-89eded48361b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:05:16.807919 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:16.807912    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ae1c3c5-f16b-4e41-b70a-64999f5ab09b-kube-api-access-j4sv2\" (UniqueName: \"kubernetes.io/projected/2ae1c3c5-f16b-4e41-b70a-64999f5ab09b-kube-api-access-j4sv2\") pod \"2ae1c3c5-f16b-4e41-b70a-64999f5ab09b\" (UID: \"2ae1c3c5-f16b-4e41-b70a-64999f5ab09b\") "
Nov 03 23:05:16.808463 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:16.808443    4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ae1c3c5-f16b-4e41-b70a-64999f5ab09b-plugins-config-volume\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:05:16.808463 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:16.808463    4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/444d998c-f1d3-46d1-8a77-89eded48361b-plugins-config-volume\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:05:16.809970 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:16.809945    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/444d998c-f1d3-46d1-8a77-89eded48361b-kube-api-access-g9bfw" pod "444d998c-f1d3-46d1-8a77-89eded48361b" (UID: "444d998c-f1d3-46d1-8a77-89eded48361b"). InnerVolumeSpecName "kube-api-access-g9bfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:05:16.810496 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:16.810476    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ae1c3c5-f16b-4e41-b70a-64999f5ab09b-kube-api-access-j4sv2" pod "2ae1c3c5-f16b-4e41-b70a-64999f5ab09b" (UID: "2ae1c3c5-f16b-4e41-b70a-64999f5ab09b"). InnerVolumeSpecName "kube-api-access-j4sv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:05:16.909277 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:16.909164    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9bfw\" (UniqueName: \"kubernetes.io/projected/444d998c-f1d3-46d1-8a77-89eded48361b-kube-api-access-g9bfw\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:05:16.909277 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:16.909196    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4sv2\" (UniqueName: \"kubernetes.io/projected/2ae1c3c5-f16b-4e41-b70a-64999f5ab09b-kube-api-access-j4sv2\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:05:17.376237 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:17.376195    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:17.482885 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:17.482847    4431 generic.go:356] "Generic (PLEG): container finished" podID="2ae1c3c5-f16b-4e41-b70a-64999f5ab09b" containerID="8740867a7cdc96212e7da8ea9e87c29dd2eea39b6f7119c7e11a579798da967b" exitCode=0
Nov 03 23:05:17.483068 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:17.482906    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-b023a2-678fcf5d79-x2g2r"
Nov 03 23:05:17.483068 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:17.482934    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-b023a2-678fcf5d79-x2g2r" event={"ID":"2ae1c3c5-f16b-4e41-b70a-64999f5ab09b","Type":"ContainerDied","Data":"8740867a7cdc96212e7da8ea9e87c29dd2eea39b6f7119c7e11a579798da967b"}
Nov 03 23:05:17.483068 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:17.482989    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-b023a2-678fcf5d79-x2g2r" event={"ID":"2ae1c3c5-f16b-4e41-b70a-64999f5ab09b","Type":"ContainerDied","Data":"7680755c303deb0aa014d258e234deaf881317f5054adcd2dc254c91e62ba7e7"}
Nov 03 23:05:17.483068 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:17.483021    4431 scope.go:122] "RemoveContainer" containerID="8740867a7cdc96212e7da8ea9e87c29dd2eea39b6f7119c7e11a579798da967b"
Nov 03 23:05:17.484581 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:17.484556    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-b023a2-f5bd44854-vjwb2" event={"ID":"444d998c-f1d3-46d1-8a77-89eded48361b","Type":"ContainerDied","Data":"3d958bc2d8a3129c1b24ba9fef68605d1d535221ce721b6dfe08de9c443c0e77"}
Nov 03 23:05:17.484706 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:17.484604    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-b023a2-f5bd44854-vjwb2"
Nov 03 23:05:17.493134 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:17.493110    4431 scope.go:122] "RemoveContainer" containerID="8740867a7cdc96212e7da8ea9e87c29dd2eea39b6f7119c7e11a579798da967b"
Nov 03 23:05:17.493455 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:05:17.493433    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8740867a7cdc96212e7da8ea9e87c29dd2eea39b6f7119c7e11a579798da967b\": container with ID starting with 8740867a7cdc96212e7da8ea9e87c29dd2eea39b6f7119c7e11a579798da967b not found: ID does not exist" containerID="8740867a7cdc96212e7da8ea9e87c29dd2eea39b6f7119c7e11a579798da967b"
Nov 03 23:05:17.493574 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:17.493464    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8740867a7cdc96212e7da8ea9e87c29dd2eea39b6f7119c7e11a579798da967b"} err="failed to get container status \"8740867a7cdc96212e7da8ea9e87c29dd2eea39b6f7119c7e11a579798da967b\": rpc error: code = NotFound desc = could not find container \"8740867a7cdc96212e7da8ea9e87c29dd2eea39b6f7119c7e11a579798da967b\": container with ID starting with 8740867a7cdc96212e7da8ea9e87c29dd2eea39b6f7119c7e11a579798da967b not found: ID does not exist"
Nov 03 23:05:17.493574 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:17.493488    4431 scope.go:122] "RemoveContainer" containerID="40694285abbdc4f03139caa5c0632d85944095a3cd816759f7dbf875a5648ccf"
Nov 03 23:05:17.534908 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:17.534880    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-b023a2-678fcf5d79-x2g2r"]
Nov 03 23:05:17.547994 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:17.547964    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-b023a2-678fcf5d79-x2g2r"]
Nov 03 23:05:17.579965 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:17.579926    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-b023a2-f5bd44854-vjwb2"]
Nov 03 23:05:17.584045 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:17.584011    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-b023a2-f5bd44854-vjwb2"]
Nov 03 23:05:18.087282 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:18.087250    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ae1c3c5-f16b-4e41-b70a-64999f5ab09b" path="/var/lib/kubelet/pods/2ae1c3c5-f16b-4e41-b70a-64999f5ab09b/volumes"
Nov 03 23:05:18.087734 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:18.087721    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="444d998c-f1d3-46d1-8a77-89eded48361b" path="/var/lib/kubelet/pods/444d998c-f1d3-46d1-8a77-89eded48361b/volumes"
Nov 03 23:05:18.684245 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:18.684196    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:18.806199 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:18.806167    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4ad693-589cf5d8f6-lgfqh"]
Nov 03 23:05:18.806735 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:18.806437    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ad693-589cf5d8f6-lgfqh" podUID="da056270-6c9e-4786-8b6e-e706e6fd3dcd" containerName="payload-processing" containerID="cri-o://db0386b4c4982e1bad405e9ca9598e262ac0b6907e690ba945d7ff58a31eb4fb" gracePeriod=30
Nov 03 23:05:18.840098 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:18.840065    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4ad693-5685dfd7bc-vbh2d"]
Nov 03 23:05:18.840535 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:18.840480    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ad693-5685dfd7bc-vbh2d" podUID="afd0d809-1add-4229-aa4f-eb5b688c2a10" containerName="payload-pre-processing" containerID="cri-o://04570e9b3f067446b147eb5613f1f8d3d466c4d2d07cdbe7cff8e4d315598d24" gracePeriod=30
Nov 03 23:05:18.986260 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:18.983052    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:19.201399 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:19.192782    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:19.240292 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:19.240263    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ad693-589cf5d8f6-lgfqh"
Nov 03 23:05:19.243834 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:19.243800    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ad693-5685dfd7bc-vbh2d"
Nov 03 23:05:19.298013 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:19.297976    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da056270-6c9e-4786-8b6e-e706e6fd3dcd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da056270-6c9e-4786-8b6e-e706e6fd3dcd-plugins-config-volume\") pod \"da056270-6c9e-4786-8b6e-e706e6fd3dcd\" (UID: \"da056270-6c9e-4786-8b6e-e706e6fd3dcd\") "
Nov 03 23:05:19.298233 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:19.298108    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da056270-6c9e-4786-8b6e-e706e6fd3dcd-kube-api-access-swps6\" (UniqueName: \"kubernetes.io/projected/da056270-6c9e-4786-8b6e-e706e6fd3dcd-kube-api-access-swps6\") pod \"da056270-6c9e-4786-8b6e-e706e6fd3dcd\" (UID: \"da056270-6c9e-4786-8b6e-e706e6fd3dcd\") "
Nov 03 23:05:19.298233 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:19.298173    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afd0d809-1add-4229-aa4f-eb5b688c2a10-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afd0d809-1add-4229-aa4f-eb5b688c2a10-plugins-config-volume\") pod \"afd0d809-1add-4229-aa4f-eb5b688c2a10\" (UID: \"afd0d809-1add-4229-aa4f-eb5b688c2a10\") "
Nov 03 23:05:19.298830 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:19.298803    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da056270-6c9e-4786-8b6e-e706e6fd3dcd-plugins-config-volume" pod "da056270-6c9e-4786-8b6e-e706e6fd3dcd" (UID: "da056270-6c9e-4786-8b6e-e706e6fd3dcd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:05:19.299141 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:19.299115    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afd0d809-1add-4229-aa4f-eb5b688c2a10-plugins-config-volume" pod "afd0d809-1add-4229-aa4f-eb5b688c2a10" (UID: "afd0d809-1add-4229-aa4f-eb5b688c2a10"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:05:19.309201 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:19.309166    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da056270-6c9e-4786-8b6e-e706e6fd3dcd-kube-api-access-swps6" pod "da056270-6c9e-4786-8b6e-e706e6fd3dcd" (UID: "da056270-6c9e-4786-8b6e-e706e6fd3dcd"). InnerVolumeSpecName "kube-api-access-swps6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:05:19.362247 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:19.362185    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:19.399363 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:19.399324    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afd0d809-1add-4229-aa4f-eb5b688c2a10-kube-api-access-bzfvh\" (UniqueName: \"kubernetes.io/projected/afd0d809-1add-4229-aa4f-eb5b688c2a10-kube-api-access-bzfvh\") pod \"afd0d809-1add-4229-aa4f-eb5b688c2a10\" (UID: \"afd0d809-1add-4229-aa4f-eb5b688c2a10\") "
Nov 03 23:05:19.400496 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:19.400464    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swps6\" (UniqueName: \"kubernetes.io/projected/da056270-6c9e-4786-8b6e-e706e6fd3dcd-kube-api-access-swps6\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:05:19.400715 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:19.400699    4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afd0d809-1add-4229-aa4f-eb5b688c2a10-plugins-config-volume\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:05:19.400834 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:19.400818    4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da056270-6c9e-4786-8b6e-e706e6fd3dcd-plugins-config-volume\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:05:19.428602 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:19.428514    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afd0d809-1add-4229-aa4f-eb5b688c2a10-kube-api-access-bzfvh" pod "afd0d809-1add-4229-aa4f-eb5b688c2a10" (UID: "afd0d809-1add-4229-aa4f-eb5b688c2a10"). InnerVolumeSpecName "kube-api-access-bzfvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:05:19.500739 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:19.500656    4431 generic.go:356] "Generic (PLEG): container finished" podID="afd0d809-1add-4229-aa4f-eb5b688c2a10" containerID="04570e9b3f067446b147eb5613f1f8d3d466c4d2d07cdbe7cff8e4d315598d24" exitCode=0
Nov 03 23:05:19.500739 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:19.500722    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ad693-5685dfd7bc-vbh2d"
Nov 03 23:05:19.500967 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:19.500733    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ad693-5685dfd7bc-vbh2d" event={"ID":"afd0d809-1add-4229-aa4f-eb5b688c2a10","Type":"ContainerDied","Data":"04570e9b3f067446b147eb5613f1f8d3d466c4d2d07cdbe7cff8e4d315598d24"}
Nov 03 23:05:19.500967 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:19.500782    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ad693-5685dfd7bc-vbh2d" event={"ID":"afd0d809-1add-4229-aa4f-eb5b688c2a10","Type":"ContainerDied","Data":"dbb4a93504c21971580a9ae50dd59b77de143f186cad32e69fb6d88fca188555"}
Nov 03 23:05:19.500967 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:19.500809    4431 scope.go:122] "RemoveContainer" containerID="04570e9b3f067446b147eb5613f1f8d3d466c4d2d07cdbe7cff8e4d315598d24"
Nov 03 23:05:19.508036 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:19.503432    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzfvh\" (UniqueName: \"kubernetes.io/projected/afd0d809-1add-4229-aa4f-eb5b688c2a10-kube-api-access-bzfvh\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:05:19.509988 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:19.508457    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:19.509988 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:19.509263    4431 generic.go:356] "Generic (PLEG): container finished" podID="da056270-6c9e-4786-8b6e-e706e6fd3dcd" containerID="db0386b4c4982e1bad405e9ca9598e262ac0b6907e690ba945d7ff58a31eb4fb" exitCode=0
Nov 03 23:05:19.509988 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:19.509385    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ad693-589cf5d8f6-lgfqh" event={"ID":"da056270-6c9e-4786-8b6e-e706e6fd3dcd","Type":"ContainerDied","Data":"db0386b4c4982e1bad405e9ca9598e262ac0b6907e690ba945d7ff58a31eb4fb"}
Nov 03 23:05:19.509988 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:19.509496    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ad693-589cf5d8f6-lgfqh" event={"ID":"da056270-6c9e-4786-8b6e-e706e6fd3dcd","Type":"ContainerDied","Data":"ab125054289681e90f0c1b40a7ec50739b4c6644ef2756570f9efaf8464ddcec"}
Nov 03 23:05:19.509988 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:19.509609    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ad693-589cf5d8f6-lgfqh"
Nov 03 23:05:19.525060 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:19.525041    4431 scope.go:122] "RemoveContainer" containerID="04570e9b3f067446b147eb5613f1f8d3d466c4d2d07cdbe7cff8e4d315598d24"
Nov 03 23:05:19.525427 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:05:19.525402    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04570e9b3f067446b147eb5613f1f8d3d466c4d2d07cdbe7cff8e4d315598d24\": container with ID starting with 04570e9b3f067446b147eb5613f1f8d3d466c4d2d07cdbe7cff8e4d315598d24 not found: ID does not exist" containerID="04570e9b3f067446b147eb5613f1f8d3d466c4d2d07cdbe7cff8e4d315598d24"
Nov 03 23:05:19.525542 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:19.525438    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04570e9b3f067446b147eb5613f1f8d3d466c4d2d07cdbe7cff8e4d315598d24"} err="failed to get container status \"04570e9b3f067446b147eb5613f1f8d3d466c4d2d07cdbe7cff8e4d315598d24\": rpc error: code = NotFound desc = could not find container \"04570e9b3f067446b147eb5613f1f8d3d466c4d2d07cdbe7cff8e4d315598d24\": container with ID starting with 04570e9b3f067446b147eb5613f1f8d3d466c4d2d07cdbe7cff8e4d315598d24 not found: ID does not exist"
Nov 03 23:05:19.525542 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:19.525461    4431 scope.go:122] "RemoveContainer" containerID="db0386b4c4982e1bad405e9ca9598e262ac0b6907e690ba945d7ff58a31eb4fb"
Nov 03 23:05:19.534388 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:19.534367    4431 scope.go:122] "RemoveContainer" containerID="db0386b4c4982e1bad405e9ca9598e262ac0b6907e690ba945d7ff58a31eb4fb"
Nov 03 23:05:19.534675 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:05:19.534653    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db0386b4c4982e1bad405e9ca9598e262ac0b6907e690ba945d7ff58a31eb4fb\": container with ID starting with db0386b4c4982e1bad405e9ca9598e262ac0b6907e690ba945d7ff58a31eb4fb not found: ID does not exist" containerID="db0386b4c4982e1bad405e9ca9598e262ac0b6907e690ba945d7ff58a31eb4fb"
Nov 03 23:05:19.534756 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:19.534684    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db0386b4c4982e1bad405e9ca9598e262ac0b6907e690ba945d7ff58a31eb4fb"} err="failed to get container status \"db0386b4c4982e1bad405e9ca9598e262ac0b6907e690ba945d7ff58a31eb4fb\": rpc error: code = NotFound desc = could not find container \"db0386b4c4982e1bad405e9ca9598e262ac0b6907e690ba945d7ff58a31eb4fb\": container with ID starting with db0386b4c4982e1bad405e9ca9598e262ac0b6907e690ba945d7ff58a31eb4fb not found: ID does not exist"
Nov 03 23:05:19.567898 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:19.567871    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4ad693-589cf5d8f6-lgfqh"]
Nov 03 23:05:19.574632 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:19.574609    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4ad693-589cf5d8f6-lgfqh"]
Nov 03 23:05:19.606600 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:19.606571    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4ad693-5685dfd7bc-vbh2d"]
Nov 03 23:05:19.611059 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:19.611033    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4ad693-5685dfd7bc-vbh2d"]
Nov 03 23:05:19.902170 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:19.902134    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:20.087643 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:20.087567    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afd0d809-1add-4229-aa4f-eb5b688c2a10" path="/var/lib/kubelet/pods/afd0d809-1add-4229-aa4f-eb5b688c2a10/volumes"
Nov 03 23:05:20.087997 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:20.087983    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da056270-6c9e-4786-8b6e-e706e6fd3dcd" path="/var/lib/kubelet/pods/da056270-6c9e-4786-8b6e-e706e6fd3dcd/volumes"
Nov 03 23:05:20.813264 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:20.813226    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w6-2a2eca/rate-test-model-2a2eca-kserve-7bf8dbc765-xm2vl"]
Nov 03 23:05:20.887278 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:20.886076    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:21.041615 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:21.041579    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:21.519829 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:21.519789    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:21.659778 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:21.659744    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:22.154772 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:22.154738    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:23.521648 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:23.521608    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:23.974721 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:23.974682    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:24.121859 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:24.121818    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:24.270567 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:24.270483    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:24.572257 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:24.572146    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:24.733821 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:24.733784    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:24.931405 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:24.931373    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:25.951277 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:25.951246    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:26.028681 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:26.028642    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-746ccb70-wxzpp"]
Nov 03 23:05:26.040491 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:26.040456    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-746ccb70-wxzpp"]
Nov 03 23:05:26.089034 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:26.088991    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ac3bb0e-6b8e-4b71-be4d-06fb3a2135b8" path="/var/lib/kubelet/pods/5ac3bb0e-6b8e-4b71-be4d-06fb3a2135b8/volumes"
Nov 03 23:05:26.576041 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:26.575998    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:26.717354 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:26.717322    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:26.840280 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:26.840166    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f"]
Nov 03 23:05:26.840735 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:26.840703    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f" podUID="a04d44db-0f1c-49c5-aa5b-069a256aa783" containerName="istio-proxy" containerID="cri-o://f427517d211dcaf69950307e501de1630bbba1fa5221ce4cc180f1ec7e063803" gracePeriod=30
Nov 03 23:05:26.852191 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:26.852163    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm"]
Nov 03 23:05:26.852533 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:26.852504    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm" podUID="acb636f9-9d36-49e8-af20-67b01ee4c095" containerName="istio-proxy" containerID="cri-o://deed00d61b410ef18d2e7a55f69970ccedd03767e6a753b3eae9534c40ecdb03" gracePeriod=30
Nov 03 23:05:26.856798 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:26.856775    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj"]
Nov 03 23:05:26.857001 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:26.856979    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj" podUID="e5ba197f-88fc-484c-8fbd-b52d1b601a99" containerName="istio-proxy" containerID="cri-o://3599fa82d00f100f265610a263318454b15f740dcbbbb7d563c97c5b2e3a0983" gracePeriod=30
Nov 03 23:05:26.907190 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:26.907127    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:27.155193 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:27.155121    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:27.296971 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:27.296937    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:27.527322 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:27.527288    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f6dcc0-7fc8d968b7-4fllg"]
Nov 03 23:05:27.527822 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:27.527809    4431 cpu_manager.go:414] "containerMap: removing container" podUID="da056270-6c9e-4786-8b6e-e706e6fd3dcd" containerName="payload-processing"
Nov 03 23:05:27.527870 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:27.527824    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da056270-6c9e-4786-8b6e-e706e6fd3dcd" containerName="payload-processing"
Nov 03 23:05:27.527870 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:27.527836    4431 cpu_manager.go:414] "containerMap: removing container" podUID="fe16bcab-2fb0-425d-961c-adc59648bace" containerName="istio-proxy"
Nov 03 23:05:27.527870 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:27.527841    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe16bcab-2fb0-425d-961c-adc59648bace" containerName="istio-proxy"
Nov 03 23:05:27.527870 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:27.527863    4431 cpu_manager.go:414] "containerMap: removing container" podUID="fe16bcab-2fb0-425d-961c-adc59648bace" containerName="istio-proxy"
Nov 03 23:05:27.527870 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:27.527869    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe16bcab-2fb0-425d-961c-adc59648bace" containerName="istio-proxy"
Nov 03 23:05:27.528043 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:27.527878    4431 cpu_manager.go:414] "containerMap: removing container" podUID="444d998c-f1d3-46d1-8a77-89eded48361b" containerName="payload-processing"
Nov 03 23:05:27.528043 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:27.527884    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="444d998c-f1d3-46d1-8a77-89eded48361b" containerName="payload-processing"
Nov 03 23:05:27.528043 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:27.527896    4431 cpu_manager.go:414] "containerMap: removing container" podUID="a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e" containerName="main"
Nov 03 23:05:27.528043 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:27.527900    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e" containerName="main"
Nov 03 23:05:27.528043 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:27.527908    4431 cpu_manager.go:414] "containerMap: removing container" podUID="61d2223e-d3b7-42ec-956d-ac7b3f94180c" containerName="payload-processing"
Nov 03 23:05:27.528043 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:27.527914    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61d2223e-d3b7-42ec-956d-ac7b3f94180c" containerName="payload-processing"
Nov 03 23:05:27.528043 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:27.527928    4431 cpu_manager.go:414] "containerMap: removing container" podUID="a1894b80-164c-4ce1-94f3-1e5011d9759e" containerName="main"
Nov 03 23:05:27.528043 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:27.527932    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1894b80-164c-4ce1-94f3-1e5011d9759e" containerName="main"
Nov 03 23:05:27.528043 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:27.527942    4431 cpu_manager.go:414] "containerMap: removing container" podUID="72eb7f3b-8510-4699-8764-72c129b39943" containerName="payload-pre-processing"
Nov 03 23:05:27.528043 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:27.527946    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72eb7f3b-8510-4699-8764-72c129b39943" containerName="payload-pre-processing"
Nov 03 23:05:27.528043 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:27.527954    4431 cpu_manager.go:414] "containerMap: removing container" podUID="9b7dd4de-ba44-4b67-a140-55d8008f5c6f" containerName="payload-processing"
Nov 03 23:05:27.528043 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:27.527959    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b7dd4de-ba44-4b67-a140-55d8008f5c6f" containerName="payload-processing"
Nov 03 23:05:27.528043 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:27.527965    4431 cpu_manager.go:414] "containerMap: removing container" podUID="8dea78d3-8945-4582-8fdb-38b0a28ad416" containerName="payload-pre-processing"
Nov 03 23:05:27.528043 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:27.527970    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dea78d3-8945-4582-8fdb-38b0a28ad416" containerName="payload-pre-processing"
Nov 03 23:05:27.528043 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:27.527978    4431 cpu_manager.go:414] "containerMap: removing container" podUID="0d83968f-285d-4804-a015-7bddf4586765" containerName="maas-api"
Nov 03 23:05:27.528043 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:27.527983    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d83968f-285d-4804-a015-7bddf4586765" containerName="maas-api"
Nov 03 23:05:27.528043 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:27.527992    4431 cpu_manager.go:414] "containerMap: removing container" podUID="2ae1c3c5-f16b-4e41-b70a-64999f5ab09b" containerName="payload-pre-processing"
Nov 03 23:05:27.528043 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:27.527997    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ae1c3c5-f16b-4e41-b70a-64999f5ab09b" containerName="payload-pre-processing"
Nov 03 23:05:27.528043 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:27.528005    4431 cpu_manager.go:414] "containerMap: removing container" podUID="afd0d809-1add-4229-aa4f-eb5b688c2a10" containerName="payload-pre-processing"
Nov 03 23:05:27.528043 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:27.528009    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afd0d809-1add-4229-aa4f-eb5b688c2a10" containerName="payload-pre-processing"
Nov 03 23:05:27.528674 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:27.528085    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1894b80-164c-4ce1-94f3-1e5011d9759e" containerName="main"
Nov 03 23:05:27.528674 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:27.528095    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="72eb7f3b-8510-4699-8764-72c129b39943" containerName="payload-pre-processing"
Nov 03 23:05:27.528674 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:27.528100    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b7dd4de-ba44-4b67-a140-55d8008f5c6f" containerName="payload-processing"
Nov 03 23:05:27.528674 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:27.528106    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dea78d3-8945-4582-8fdb-38b0a28ad416" containerName="payload-pre-processing"
Nov 03 23:05:27.528674 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:27.528111    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d83968f-285d-4804-a015-7bddf4586765" containerName="maas-api"
Nov 03 23:05:27.528674 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:27.528118    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ae1c3c5-f16b-4e41-b70a-64999f5ab09b" containerName="payload-pre-processing"
Nov 03 23:05:27.528674 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:27.528124    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="afd0d809-1add-4229-aa4f-eb5b688c2a10" containerName="payload-pre-processing"
Nov 03 23:05:27.528674 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:27.528131    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="da056270-6c9e-4786-8b6e-e706e6fd3dcd" containerName="payload-processing"
Nov 03 23:05:27.528674 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:27.528137    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe16bcab-2fb0-425d-961c-adc59648bace" containerName="istio-proxy"
Nov 03 23:05:27.528674 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:27.528144    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe16bcab-2fb0-425d-961c-adc59648bace" containerName="istio-proxy"
Nov 03 23:05:27.528674 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:27.528151    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="444d998c-f1d3-46d1-8a77-89eded48361b" containerName="payload-processing"
Nov 03 23:05:27.528674 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:27.528157    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="a762e2f1-30fd-4db7-8f5b-67aa1cc6a98e" containerName="main"
Nov 03 23:05:27.528674 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:27.528162    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="61d2223e-d3b7-42ec-956d-ac7b3f94180c" containerName="payload-processing"
Nov 03 23:05:27.533436 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:27.533415    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f6dcc0-7fc8d968b7-4fllg"
Nov 03 23:05:27.548688 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:27.548657    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-f6dcc0\""
Nov 03 23:05:27.548688 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:27.548665    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-f6dcc0-dockercfg-hdc5f\""
Nov 03 23:05:27.550734 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:27.550712    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f6dcc0-7fc8d968b7-4fllg"]
Nov 03 23:05:27.612068 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:27.612024    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9z5h\" (UniqueName: \"kubernetes.io/projected/a125bb41-5bfd-4516-b7dc-971e38e6e8a3-kube-api-access-b9z5h\") pod \"payload-pre-processing-e2e-ipp-b-f6dcc0-7fc8d968b7-4fllg\" (UID: \"a125bb41-5bfd-4516-b7dc-971e38e6e8a3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f6dcc0-7fc8d968b7-4fllg"
Nov 03 23:05:27.612321 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:27.612298    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a125bb41-5bfd-4516-b7dc-971e38e6e8a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f6dcc0-7fc8d968b7-4fllg\" (UID: \"a125bb41-5bfd-4516-b7dc-971e38e6e8a3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f6dcc0-7fc8d968b7-4fllg"
Nov 03 23:05:27.704344 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:27.704304    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f6dcc0-6b5944664-gzcth"]
Nov 03 23:05:27.708464 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:27.708443    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f6dcc0-6b5944664-gzcth"
Nov 03 23:05:27.713384 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:27.713357    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9z5h\" (UniqueName: \"kubernetes.io/projected/a125bb41-5bfd-4516-b7dc-971e38e6e8a3-kube-api-access-b9z5h\") pod \"payload-pre-processing-e2e-ipp-b-f6dcc0-7fc8d968b7-4fllg\" (UID: \"a125bb41-5bfd-4516-b7dc-971e38e6e8a3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f6dcc0-7fc8d968b7-4fllg"
Nov 03 23:05:27.713508 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:27.713495    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a125bb41-5bfd-4516-b7dc-971e38e6e8a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f6dcc0-7fc8d968b7-4fllg\" (UID: \"a125bb41-5bfd-4516-b7dc-971e38e6e8a3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f6dcc0-7fc8d968b7-4fllg"
Nov 03 23:05:27.714155 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:27.714139    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a125bb41-5bfd-4516-b7dc-971e38e6e8a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f6dcc0-7fc8d968b7-4fllg\" (UID: \"a125bb41-5bfd-4516-b7dc-971e38e6e8a3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f6dcc0-7fc8d968b7-4fllg"
Nov 03 23:05:27.725822 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:27.725798    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f6dcc0-6b5944664-gzcth"]
Nov 03 23:05:27.738297 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:27.738270    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9z5h\" (UniqueName: \"kubernetes.io/projected/a125bb41-5bfd-4516-b7dc-971e38e6e8a3-kube-api-access-b9z5h\") pod \"payload-pre-processing-e2e-ipp-b-f6dcc0-7fc8d968b7-4fllg\" (UID: \"a125bb41-5bfd-4516-b7dc-971e38e6e8a3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f6dcc0-7fc8d968b7-4fllg"
Nov 03 23:05:27.815159 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:27.815080    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7gp6\" (UniqueName: \"kubernetes.io/projected/e700154c-e286-4bb3-87e1-cd0e965cbdc4-kube-api-access-n7gp6\") pod \"payload-processing-e2e-ipp-b-f6dcc0-6b5944664-gzcth\" (UID: \"e700154c-e286-4bb3-87e1-cd0e965cbdc4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f6dcc0-6b5944664-gzcth"
Nov 03 23:05:27.815341 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:27.815193    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e700154c-e286-4bb3-87e1-cd0e965cbdc4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f6dcc0-6b5944664-gzcth\" (UID: \"e700154c-e286-4bb3-87e1-cd0e965cbdc4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f6dcc0-6b5944664-gzcth"
Nov 03 23:05:27.847922 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:27.847886    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f6dcc0-7fc8d968b7-4fllg"
Nov 03 23:05:27.918568 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:27.917963    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7gp6\" (UniqueName: \"kubernetes.io/projected/e700154c-e286-4bb3-87e1-cd0e965cbdc4-kube-api-access-n7gp6\") pod \"payload-processing-e2e-ipp-b-f6dcc0-6b5944664-gzcth\" (UID: \"e700154c-e286-4bb3-87e1-cd0e965cbdc4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f6dcc0-6b5944664-gzcth"
Nov 03 23:05:27.918568 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:27.918081    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e700154c-e286-4bb3-87e1-cd0e965cbdc4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f6dcc0-6b5944664-gzcth\" (UID: \"e700154c-e286-4bb3-87e1-cd0e965cbdc4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f6dcc0-6b5944664-gzcth"
Nov 03 23:05:27.919158 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:27.919127    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e700154c-e286-4bb3-87e1-cd0e965cbdc4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f6dcc0-6b5944664-gzcth\" (UID: \"e700154c-e286-4bb3-87e1-cd0e965cbdc4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f6dcc0-6b5944664-gzcth"
Nov 03 23:05:27.942497 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:27.942276    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7gp6\" (UniqueName: \"kubernetes.io/projected/e700154c-e286-4bb3-87e1-cd0e965cbdc4-kube-api-access-n7gp6\") pod \"payload-processing-e2e-ipp-b-f6dcc0-6b5944664-gzcth\" (UID: \"e700154c-e286-4bb3-87e1-cd0e965cbdc4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f6dcc0-6b5944664-gzcth"
Nov 03 23:05:28.023180 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:28.023145    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f6dcc0-6b5944664-gzcth"
Nov 03 23:05:28.148694 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:28.148665    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f6dcc0-7fc8d968b7-4fllg"]
Nov 03 23:05:28.151614 ip-10-0-139-73 kubenswrapper[4431]: W1103 23:05:28.151581    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda125bb41_5bfd_4516_b7dc_971e38e6e8a3.slice/crio-593f73e7c3af3b8dd90db3e2877b9398b48e08f9d5b763dbd5ac189d3e250b72 WatchSource:0}: Error finding container 593f73e7c3af3b8dd90db3e2877b9398b48e08f9d5b763dbd5ac189d3e250b72: Status 404 returned error can't find the container with id 593f73e7c3af3b8dd90db3e2877b9398b48e08f9d5b763dbd5ac189d3e250b72
Nov 03 23:05:28.244717 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:28.244682    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:28.325845 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:28.325772    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f6dcc0-6b5944664-gzcth"]
Nov 03 23:05:28.327950 ip-10-0-139-73 kubenswrapper[4431]: W1103 23:05:28.327909    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode700154c_e286_4bb3_87e1_cd0e965cbdc4.slice/crio-f6cdc8349541e9097b65ba947cac56c02bbe01923ae1dec147586f073cf32d8e WatchSource:0}: Error finding container f6cdc8349541e9097b65ba947cac56c02bbe01923ae1dec147586f073cf32d8e: Status 404 returned error can't find the container with id f6cdc8349541e9097b65ba947cac56c02bbe01923ae1dec147586f073cf32d8e
Nov 03 23:05:28.577862 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:28.577776    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f6dcc0-6b5944664-gzcth" event={"ID":"e700154c-e286-4bb3-87e1-cd0e965cbdc4","Type":"ContainerStarted","Data":"5208b62e81ac2fa2ad9ac4c6945a781a0fc472fcec0c455fb5a569e059f265bb"}
Nov 03 23:05:28.577862 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:28.577854    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f6dcc0-6b5944664-gzcth" event={"ID":"e700154c-e286-4bb3-87e1-cd0e965cbdc4","Type":"ContainerStarted","Data":"f6cdc8349541e9097b65ba947cac56c02bbe01923ae1dec147586f073cf32d8e"}
Nov 03 23:05:28.578114 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:28.577998    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f6dcc0-6b5944664-gzcth"
Nov 03 23:05:28.579667 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:28.579637    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f6dcc0-7fc8d968b7-4fllg" event={"ID":"a125bb41-5bfd-4516-b7dc-971e38e6e8a3","Type":"ContainerStarted","Data":"4066f46b0d6317a8d07f6ccdad3632a9f086c5c7bb72612d64c2d57ed45ddad2"}
Nov 03 23:05:28.579814 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:28.579689    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f6dcc0-7fc8d968b7-4fllg" event={"ID":"a125bb41-5bfd-4516-b7dc-971e38e6e8a3","Type":"ContainerStarted","Data":"593f73e7c3af3b8dd90db3e2877b9398b48e08f9d5b763dbd5ac189d3e250b72"}
Nov 03 23:05:28.579814 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:28.579732    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f6dcc0-7fc8d968b7-4fllg"
Nov 03 23:05:28.621483 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:28.621409    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-f6dcc0-6b5944664-gzcth" podStartSLOduration=1.621391408 podStartE2EDuration="1.621391408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:05:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:05:28.61814924 +0000 UTC m=+3152.644005101" watchObservedRunningTime="2026-11-03 23:05:28.621391408 +0000 UTC m=+3152.647247270"
Nov 03 23:05:28.659399 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:28.659337    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f6dcc0-7fc8d968b7-4fllg" podStartSLOduration=1.659321083 podStartE2EDuration="1.659321083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:05:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:05:28.659022871 +0000 UTC m=+3152.684878733" watchObservedRunningTime="2026-11-03 23:05:28.659321083 +0000 UTC m=+3152.685176946"
Nov 03 23:05:28.971001 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:28.970964    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:29.116662 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:29.116608    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:29.247347 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:29.247243    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9k6cb"]
Nov 03 23:05:29.247843 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:29.247536    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9k6cb" podUID="e568c172-7301-4d7f-9499-6c3a1619c9dd" containerName="istio-proxy" containerID="cri-o://9dcc65475817e4e163a86b4a88c69cd79305fa4f7c6b6739cb614a0aa7b4e416" gracePeriod=30
Nov 03 23:05:29.327555 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:29.327517    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:30.047611 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:30.047576    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:30.722117 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:30.722077    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:31.260730 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:05:31.260692    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 23:05:31.525894 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:31.525810    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:31.878478 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:31.878452    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:32.025561 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.025475    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:32.192423 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.192260    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:32.229866 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.229841    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f"
Nov 03 23:05:32.277507 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.277480    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm"
Nov 03 23:05:32.280797 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.280773    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj"
Nov 03 23:05:32.336327 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.336294    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:32.374702 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.374616    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5ba197f-88fc-484c-8fbd-b52d1b601a99-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5ba197f-88fc-484c-8fbd-b52d1b601a99-istiod-ca-cert\") pod \"e5ba197f-88fc-484c-8fbd-b52d1b601a99\" (UID: \"e5ba197f-88fc-484c-8fbd-b52d1b601a99\") "
Nov 03 23:05:32.374702 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.374664    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5ba197f-88fc-484c-8fbd-b52d1b601a99-istio-token\" (UniqueName: \"kubernetes.io/projected/e5ba197f-88fc-484c-8fbd-b52d1b601a99-istio-token\") pod \"e5ba197f-88fc-484c-8fbd-b52d1b601a99\" (UID: \"e5ba197f-88fc-484c-8fbd-b52d1b601a99\") "
Nov 03 23:05:32.374702 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.374694    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5ba197f-88fc-484c-8fbd-b52d1b601a99-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5ba197f-88fc-484c-8fbd-b52d1b601a99-istio-data\") pod \"e5ba197f-88fc-484c-8fbd-b52d1b601a99\" (UID: \"e5ba197f-88fc-484c-8fbd-b52d1b601a99\") "
Nov 03 23:05:32.375066 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.374718    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a04d44db-0f1c-49c5-aa5b-069a256aa783-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a04d44db-0f1c-49c5-aa5b-069a256aa783-istiod-ca-cert\") pod \"a04d44db-0f1c-49c5-aa5b-069a256aa783\" (UID: \"a04d44db-0f1c-49c5-aa5b-069a256aa783\") "
Nov 03 23:05:32.375066 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.374740    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acb636f9-9d36-49e8-af20-67b01ee4c095-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acb636f9-9d36-49e8-af20-67b01ee4c095-istiod-ca-cert\") pod \"acb636f9-9d36-49e8-af20-67b01ee4c095\" (UID: \"acb636f9-9d36-49e8-af20-67b01ee4c095\") "
Nov 03 23:05:32.375066 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.374771    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e5ba197f-88fc-484c-8fbd-b52d1b601a99-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5ba197f-88fc-484c-8fbd-b52d1b601a99-istio-podinfo\") pod \"e5ba197f-88fc-484c-8fbd-b52d1b601a99\" (UID: \"e5ba197f-88fc-484c-8fbd-b52d1b601a99\") "
Nov 03 23:05:32.375066 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.374802    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a04d44db-0f1c-49c5-aa5b-069a256aa783-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a04d44db-0f1c-49c5-aa5b-069a256aa783-istio-envoy\") pod \"a04d44db-0f1c-49c5-aa5b-069a256aa783\" (UID: \"a04d44db-0f1c-49c5-aa5b-069a256aa783\") "
Nov 03 23:05:32.375066 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.374858    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a04d44db-0f1c-49c5-aa5b-069a256aa783-istio-token\" (UniqueName: \"kubernetes.io/projected/a04d44db-0f1c-49c5-aa5b-069a256aa783-istio-token\") pod \"a04d44db-0f1c-49c5-aa5b-069a256aa783\" (UID: \"a04d44db-0f1c-49c5-aa5b-069a256aa783\") "
Nov 03 23:05:32.375066 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.374903    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5ba197f-88fc-484c-8fbd-b52d1b601a99-kube-api-access-47vq2\" (UniqueName: \"kubernetes.io/projected/e5ba197f-88fc-484c-8fbd-b52d1b601a99-kube-api-access-47vq2\") pod \"e5ba197f-88fc-484c-8fbd-b52d1b601a99\" (UID: \"e5ba197f-88fc-484c-8fbd-b52d1b601a99\") "
Nov 03 23:05:32.375066 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.374951    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acb636f9-9d36-49e8-af20-67b01ee4c095-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acb636f9-9d36-49e8-af20-67b01ee4c095-workload-certs\") pod \"acb636f9-9d36-49e8-af20-67b01ee4c095\" (UID: \"acb636f9-9d36-49e8-af20-67b01ee4c095\") "
Nov 03 23:05:32.375066 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.375010    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a04d44db-0f1c-49c5-aa5b-069a256aa783-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a04d44db-0f1c-49c5-aa5b-069a256aa783-workload-certs\") pod \"a04d44db-0f1c-49c5-aa5b-069a256aa783\" (UID: \"a04d44db-0f1c-49c5-aa5b-069a256aa783\") "
Nov 03 23:05:32.375066 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.375042    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5ba197f-88fc-484c-8fbd-b52d1b601a99-istio-data" pod "e5ba197f-88fc-484c-8fbd-b52d1b601a99" (UID: "e5ba197f-88fc-484c-8fbd-b52d1b601a99"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:05:32.375066 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.375059    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5ba197f-88fc-484c-8fbd-b52d1b601a99-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ba197f-88fc-484c-8fbd-b52d1b601a99-workload-socket\") pod \"e5ba197f-88fc-484c-8fbd-b52d1b601a99\" (UID: \"e5ba197f-88fc-484c-8fbd-b52d1b601a99\") "
Nov 03 23:05:32.375649 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.375078    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5ba197f-88fc-484c-8fbd-b52d1b601a99-istiod-ca-cert" pod "e5ba197f-88fc-484c-8fbd-b52d1b601a99" (UID: "e5ba197f-88fc-484c-8fbd-b52d1b601a99"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:05:32.375649 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.375088    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5ba197f-88fc-484c-8fbd-b52d1b601a99-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5ba197f-88fc-484c-8fbd-b52d1b601a99-workload-certs\") pod \"e5ba197f-88fc-484c-8fbd-b52d1b601a99\" (UID: \"e5ba197f-88fc-484c-8fbd-b52d1b601a99\") "
Nov 03 23:05:32.375649 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.375155    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/acb636f9-9d36-49e8-af20-67b01ee4c095-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acb636f9-9d36-49e8-af20-67b01ee4c095-istio-podinfo\") pod \"acb636f9-9d36-49e8-af20-67b01ee4c095\" (UID: \"acb636f9-9d36-49e8-af20-67b01ee4c095\") "
Nov 03 23:05:32.375649 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.375190    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a04d44db-0f1c-49c5-aa5b-069a256aa783-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a04d44db-0f1c-49c5-aa5b-069a256aa783-workload-socket\") pod \"a04d44db-0f1c-49c5-aa5b-069a256aa783\" (UID: \"a04d44db-0f1c-49c5-aa5b-069a256aa783\") "
Nov 03 23:05:32.375649 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.375398    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a04d44db-0f1c-49c5-aa5b-069a256aa783-kube-api-access-ss9vx\" (UniqueName: \"kubernetes.io/projected/a04d44db-0f1c-49c5-aa5b-069a256aa783-kube-api-access-ss9vx\") pod \"a04d44db-0f1c-49c5-aa5b-069a256aa783\" (UID: \"a04d44db-0f1c-49c5-aa5b-069a256aa783\") "
Nov 03 23:05:32.375649 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.375446    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5ba197f-88fc-484c-8fbd-b52d1b601a99-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5ba197f-88fc-484c-8fbd-b52d1b601a99-istio-envoy\") pod \"e5ba197f-88fc-484c-8fbd-b52d1b601a99\" (UID: \"e5ba197f-88fc-484c-8fbd-b52d1b601a99\") "
Nov 03 23:05:32.375649 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.375495    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acb636f9-9d36-49e8-af20-67b01ee4c095-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acb636f9-9d36-49e8-af20-67b01ee4c095-istio-data\") pod \"acb636f9-9d36-49e8-af20-67b01ee4c095\" (UID: \"acb636f9-9d36-49e8-af20-67b01ee4c095\") "
Nov 03 23:05:32.375649 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.375566    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acb636f9-9d36-49e8-af20-67b01ee4c095-istio-token\" (UniqueName: \"kubernetes.io/projected/acb636f9-9d36-49e8-af20-67b01ee4c095-istio-token\") pod \"acb636f9-9d36-49e8-af20-67b01ee4c095\" (UID: \"acb636f9-9d36-49e8-af20-67b01ee4c095\") "
Nov 03 23:05:32.375649 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.375602    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acb636f9-9d36-49e8-af20-67b01ee4c095-kube-api-access-pp6m5\" (UniqueName: \"kubernetes.io/projected/acb636f9-9d36-49e8-af20-67b01ee4c095-kube-api-access-pp6m5\") pod \"acb636f9-9d36-49e8-af20-67b01ee4c095\" (UID: \"acb636f9-9d36-49e8-af20-67b01ee4c095\") "
Nov 03 23:05:32.376169 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.375663    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a04d44db-0f1c-49c5-aa5b-069a256aa783-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a04d44db-0f1c-49c5-aa5b-069a256aa783-credential-socket\") pod \"a04d44db-0f1c-49c5-aa5b-069a256aa783\" (UID: \"a04d44db-0f1c-49c5-aa5b-069a256aa783\") "
Nov 03 23:05:32.376169 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.375694    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a04d44db-0f1c-49c5-aa5b-069a256aa783-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a04d44db-0f1c-49c5-aa5b-069a256aa783-istio-data\") pod \"a04d44db-0f1c-49c5-aa5b-069a256aa783\" (UID: \"a04d44db-0f1c-49c5-aa5b-069a256aa783\") "
Nov 03 23:05:32.376169 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.375735    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acb636f9-9d36-49e8-af20-67b01ee4c095-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acb636f9-9d36-49e8-af20-67b01ee4c095-workload-socket\") pod \"acb636f9-9d36-49e8-af20-67b01ee4c095\" (UID: \"acb636f9-9d36-49e8-af20-67b01ee4c095\") "
Nov 03 23:05:32.376169 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.375792    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5ba197f-88fc-484c-8fbd-b52d1b601a99-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ba197f-88fc-484c-8fbd-b52d1b601a99-credential-socket\") pod \"e5ba197f-88fc-484c-8fbd-b52d1b601a99\" (UID: \"e5ba197f-88fc-484c-8fbd-b52d1b601a99\") "
Nov 03 23:05:32.376169 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.375843    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a04d44db-0f1c-49c5-aa5b-069a256aa783-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a04d44db-0f1c-49c5-aa5b-069a256aa783-istio-podinfo\") pod \"a04d44db-0f1c-49c5-aa5b-069a256aa783\" (UID: \"a04d44db-0f1c-49c5-aa5b-069a256aa783\") "
Nov 03 23:05:32.376169 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.375890    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acb636f9-9d36-49e8-af20-67b01ee4c095-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acb636f9-9d36-49e8-af20-67b01ee4c095-istio-envoy\") pod \"acb636f9-9d36-49e8-af20-67b01ee4c095\" (UID: \"acb636f9-9d36-49e8-af20-67b01ee4c095\") "
Nov 03 23:05:32.376169 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.375922    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acb636f9-9d36-49e8-af20-67b01ee4c095-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acb636f9-9d36-49e8-af20-67b01ee4c095-credential-socket\") pod \"acb636f9-9d36-49e8-af20-67b01ee4c095\" (UID: \"acb636f9-9d36-49e8-af20-67b01ee4c095\") "
Nov 03 23:05:32.376169 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.375154    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a04d44db-0f1c-49c5-aa5b-069a256aa783-istiod-ca-cert" pod "a04d44db-0f1c-49c5-aa5b-069a256aa783" (UID: "a04d44db-0f1c-49c5-aa5b-069a256aa783"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:05:32.376169 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.375179    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acb636f9-9d36-49e8-af20-67b01ee4c095-istiod-ca-cert" pod "acb636f9-9d36-49e8-af20-67b01ee4c095" (UID: "acb636f9-9d36-49e8-af20-67b01ee4c095"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:05:32.376169 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.375328    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5ba197f-88fc-484c-8fbd-b52d1b601a99-workload-certs" pod "e5ba197f-88fc-484c-8fbd-b52d1b601a99" (UID: "e5ba197f-88fc-484c-8fbd-b52d1b601a99"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:05:32.376169 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.375519    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a04d44db-0f1c-49c5-aa5b-069a256aa783-workload-certs" pod "a04d44db-0f1c-49c5-aa5b-069a256aa783" (UID: "a04d44db-0f1c-49c5-aa5b-069a256aa783"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:05:32.376169 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.375552    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5ba197f-88fc-484c-8fbd-b52d1b601a99-workload-socket" pod "e5ba197f-88fc-484c-8fbd-b52d1b601a99" (UID: "e5ba197f-88fc-484c-8fbd-b52d1b601a99"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:05:32.376169 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.375706    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acb636f9-9d36-49e8-af20-67b01ee4c095-workload-certs" pod "acb636f9-9d36-49e8-af20-67b01ee4c095" (UID: "acb636f9-9d36-49e8-af20-67b01ee4c095"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:05:32.377052 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.376317    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a04d44db-0f1c-49c5-aa5b-069a256aa783-credential-socket" pod "a04d44db-0f1c-49c5-aa5b-069a256aa783" (UID: "a04d44db-0f1c-49c5-aa5b-069a256aa783"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:05:32.377052 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.376349    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a04d44db-0f1c-49c5-aa5b-069a256aa783-istio-data" pod "a04d44db-0f1c-49c5-aa5b-069a256aa783" (UID: "a04d44db-0f1c-49c5-aa5b-069a256aa783"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:05:32.377052 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.376768    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acb636f9-9d36-49e8-af20-67b01ee4c095-credential-socket" pod "acb636f9-9d36-49e8-af20-67b01ee4c095" (UID: "acb636f9-9d36-49e8-af20-67b01ee4c095"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:05:32.377452 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.377430    4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5ba197f-88fc-484c-8fbd-b52d1b601a99-istiod-ca-cert\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:05:32.377568 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.377458    4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5ba197f-88fc-484c-8fbd-b52d1b601a99-istio-data\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:05:32.377568 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.377472    4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a04d44db-0f1c-49c5-aa5b-069a256aa783-istiod-ca-cert\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:05:32.377568 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.377486    4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acb636f9-9d36-49e8-af20-67b01ee4c095-istiod-ca-cert\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:05:32.377568 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.377499    4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acb636f9-9d36-49e8-af20-67b01ee4c095-workload-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:05:32.377568 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.377513    4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a04d44db-0f1c-49c5-aa5b-069a256aa783-workload-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:05:32.377568 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.377526    4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ba197f-88fc-484c-8fbd-b52d1b601a99-workload-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:05:32.377568 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.377540    4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5ba197f-88fc-484c-8fbd-b52d1b601a99-workload-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:05:32.377568 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.377554    4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a04d44db-0f1c-49c5-aa5b-069a256aa783-credential-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:05:32.377568 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.377568    4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a04d44db-0f1c-49c5-aa5b-069a256aa783-istio-data\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:05:32.378309 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.377587    4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acb636f9-9d36-49e8-af20-67b01ee4c095-credential-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:05:32.378309 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.377726    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a04d44db-0f1c-49c5-aa5b-069a256aa783-workload-socket" pod "a04d44db-0f1c-49c5-aa5b-069a256aa783" (UID: "a04d44db-0f1c-49c5-aa5b-069a256aa783"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:05:32.378309 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.378136    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5ba197f-88fc-484c-8fbd-b52d1b601a99-credential-socket" pod "e5ba197f-88fc-484c-8fbd-b52d1b601a99" (UID: "e5ba197f-88fc-484c-8fbd-b52d1b601a99"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:05:32.378550 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.378396    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acb636f9-9d36-49e8-af20-67b01ee4c095-istio-data" pod "acb636f9-9d36-49e8-af20-67b01ee4c095" (UID: "acb636f9-9d36-49e8-af20-67b01ee4c095"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:05:32.378550 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.378499    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acb636f9-9d36-49e8-af20-67b01ee4c095-workload-socket" pod "acb636f9-9d36-49e8-af20-67b01ee4c095" (UID: "acb636f9-9d36-49e8-af20-67b01ee4c095"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:05:32.378893 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.378853    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5ba197f-88fc-484c-8fbd-b52d1b601a99-kube-api-access-47vq2" pod "e5ba197f-88fc-484c-8fbd-b52d1b601a99" (UID: "e5ba197f-88fc-484c-8fbd-b52d1b601a99"). InnerVolumeSpecName "kube-api-access-47vq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:05:32.379612 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.379491    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/acb636f9-9d36-49e8-af20-67b01ee4c095-istio-podinfo" pod "acb636f9-9d36-49e8-af20-67b01ee4c095" (UID: "acb636f9-9d36-49e8-af20-67b01ee4c095"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 23:05:32.379612 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.379559    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acb636f9-9d36-49e8-af20-67b01ee4c095-istio-envoy" pod "acb636f9-9d36-49e8-af20-67b01ee4c095" (UID: "acb636f9-9d36-49e8-af20-67b01ee4c095"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:05:32.379612 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.379573    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5ba197f-88fc-484c-8fbd-b52d1b601a99-istio-podinfo" pod "e5ba197f-88fc-484c-8fbd-b52d1b601a99" (UID: "e5ba197f-88fc-484c-8fbd-b52d1b601a99"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 23:05:32.379612 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.379587    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a04d44db-0f1c-49c5-aa5b-069a256aa783-istio-token" pod "a04d44db-0f1c-49c5-aa5b-069a256aa783" (UID: "a04d44db-0f1c-49c5-aa5b-069a256aa783"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:05:32.379912 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.379626    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a04d44db-0f1c-49c5-aa5b-069a256aa783-kube-api-access-ss9vx" pod "a04d44db-0f1c-49c5-aa5b-069a256aa783" (UID: "a04d44db-0f1c-49c5-aa5b-069a256aa783"). InnerVolumeSpecName "kube-api-access-ss9vx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:05:32.379912 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.379829    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acb636f9-9d36-49e8-af20-67b01ee4c095-istio-token" pod "acb636f9-9d36-49e8-af20-67b01ee4c095" (UID: "acb636f9-9d36-49e8-af20-67b01ee4c095"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:05:32.379912 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.379859    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5ba197f-88fc-484c-8fbd-b52d1b601a99-istio-token" pod "e5ba197f-88fc-484c-8fbd-b52d1b601a99" (UID: "e5ba197f-88fc-484c-8fbd-b52d1b601a99"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:05:32.380413 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.380391    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a04d44db-0f1c-49c5-aa5b-069a256aa783-istio-podinfo" pod "a04d44db-0f1c-49c5-aa5b-069a256aa783" (UID: "a04d44db-0f1c-49c5-aa5b-069a256aa783"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 23:05:32.380617 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.380598    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acb636f9-9d36-49e8-af20-67b01ee4c095-kube-api-access-pp6m5" pod "acb636f9-9d36-49e8-af20-67b01ee4c095" (UID: "acb636f9-9d36-49e8-af20-67b01ee4c095"). InnerVolumeSpecName "kube-api-access-pp6m5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:05:32.380760 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.380741    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a04d44db-0f1c-49c5-aa5b-069a256aa783-istio-envoy" pod "a04d44db-0f1c-49c5-aa5b-069a256aa783" (UID: "a04d44db-0f1c-49c5-aa5b-069a256aa783"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:05:32.381076 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.381052    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5ba197f-88fc-484c-8fbd-b52d1b601a99-istio-envoy" pod "e5ba197f-88fc-484c-8fbd-b52d1b601a99" (UID: "e5ba197f-88fc-484c-8fbd-b52d1b601a99"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:05:32.478602 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.478568    4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acb636f9-9d36-49e8-af20-67b01ee4c095-workload-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:05:32.478602 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.478598    4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ba197f-88fc-484c-8fbd-b52d1b601a99-credential-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:05:32.478791 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.478609    4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a04d44db-0f1c-49c5-aa5b-069a256aa783-istio-podinfo\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:05:32.478791 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.478620    4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acb636f9-9d36-49e8-af20-67b01ee4c095-istio-envoy\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:05:32.478791 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.478634    4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5ba197f-88fc-484c-8fbd-b52d1b601a99-istio-token\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:05:32.478791 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.478644    4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5ba197f-88fc-484c-8fbd-b52d1b601a99-istio-podinfo\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:05:32.478791 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.478651    4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a04d44db-0f1c-49c5-aa5b-069a256aa783-istio-envoy\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:05:32.478791 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.478658    4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a04d44db-0f1c-49c5-aa5b-069a256aa783-istio-token\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:05:32.478791 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.478666    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47vq2\" (UniqueName: \"kubernetes.io/projected/e5ba197f-88fc-484c-8fbd-b52d1b601a99-kube-api-access-47vq2\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:05:32.478791 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.478674    4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acb636f9-9d36-49e8-af20-67b01ee4c095-istio-podinfo\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:05:32.478791 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.478683    4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a04d44db-0f1c-49c5-aa5b-069a256aa783-workload-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:05:32.478791 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.478692    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ss9vx\" (UniqueName: \"kubernetes.io/projected/a04d44db-0f1c-49c5-aa5b-069a256aa783-kube-api-access-ss9vx\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:05:32.478791 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.478701    4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5ba197f-88fc-484c-8fbd-b52d1b601a99-istio-envoy\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:05:32.478791 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.478714    4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acb636f9-9d36-49e8-af20-67b01ee4c095-istio-data\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:05:32.478791 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.478722    4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acb636f9-9d36-49e8-af20-67b01ee4c095-istio-token\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:05:32.478791 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.478730    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pp6m5\" (UniqueName: \"kubernetes.io/projected/acb636f9-9d36-49e8-af20-67b01ee4c095-kube-api-access-pp6m5\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:05:32.607778 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.607740    4431 generic.go:356] "Generic (PLEG): container finished" podID="acb636f9-9d36-49e8-af20-67b01ee4c095" containerID="deed00d61b410ef18d2e7a55f69970ccedd03767e6a753b3eae9534c40ecdb03" exitCode=0
Nov 03 23:05:32.607953 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.607790    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm"
Nov 03 23:05:32.607953 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.607824    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm" event={"ID":"acb636f9-9d36-49e8-af20-67b01ee4c095","Type":"ContainerDied","Data":"deed00d61b410ef18d2e7a55f69970ccedd03767e6a753b3eae9534c40ecdb03"}
Nov 03 23:05:32.607953 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.607889    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm" event={"ID":"acb636f9-9d36-49e8-af20-67b01ee4c095","Type":"ContainerDied","Data":"4f3c746ea0cd262b5b56ec645c873c70432cd1f7a44713e364c2c231e163138d"}
Nov 03 23:05:32.607953 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.607924    4431 scope.go:122] "RemoveContainer" containerID="deed00d61b410ef18d2e7a55f69970ccedd03767e6a753b3eae9534c40ecdb03"
Nov 03 23:05:32.609289 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.609260    4431 generic.go:356] "Generic (PLEG): container finished" podID="a04d44db-0f1c-49c5-aa5b-069a256aa783" containerID="f427517d211dcaf69950307e501de1630bbba1fa5221ce4cc180f1ec7e063803" exitCode=0
Nov 03 23:05:32.609415 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.609309    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f"
Nov 03 23:05:32.609415 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.609351    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f" event={"ID":"a04d44db-0f1c-49c5-aa5b-069a256aa783","Type":"ContainerDied","Data":"f427517d211dcaf69950307e501de1630bbba1fa5221ce4cc180f1ec7e063803"}
Nov 03 23:05:32.609415 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.609400    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f" event={"ID":"a04d44db-0f1c-49c5-aa5b-069a256aa783","Type":"ContainerDied","Data":"0edd7333a66e1481f4305ec168085f6b68fe2b62efdeb4b5115db0f6d48ede41"}
Nov 03 23:05:32.619576 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.619550    4431 generic.go:356] "Generic (PLEG): container finished" podID="e5ba197f-88fc-484c-8fbd-b52d1b601a99" containerID="3599fa82d00f100f265610a263318454b15f740dcbbbb7d563c97c5b2e3a0983" exitCode=0
Nov 03 23:05:32.619714 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.619591    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj"
Nov 03 23:05:32.619714 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.619636    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj" event={"ID":"e5ba197f-88fc-484c-8fbd-b52d1b601a99","Type":"ContainerDied","Data":"3599fa82d00f100f265610a263318454b15f740dcbbbb7d563c97c5b2e3a0983"}
Nov 03 23:05:32.619714 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.619691    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj" event={"ID":"e5ba197f-88fc-484c-8fbd-b52d1b601a99","Type":"ContainerDied","Data":"f6188fdfda650dd1396a55bcecbc5d84aace305e8f0cd7a9bb86a91037718ef6"}
Nov 03 23:05:32.621660 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.621626    4431 scope.go:122] "RemoveContainer" containerID="deed00d61b410ef18d2e7a55f69970ccedd03767e6a753b3eae9534c40ecdb03"
Nov 03 23:05:32.622478 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:05:32.622340    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"deed00d61b410ef18d2e7a55f69970ccedd03767e6a753b3eae9534c40ecdb03\": container with ID starting with deed00d61b410ef18d2e7a55f69970ccedd03767e6a753b3eae9534c40ecdb03 not found: ID does not exist" containerID="deed00d61b410ef18d2e7a55f69970ccedd03767e6a753b3eae9534c40ecdb03"
Nov 03 23:05:32.622478 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.622375    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"deed00d61b410ef18d2e7a55f69970ccedd03767e6a753b3eae9534c40ecdb03"} err="failed to get container status \"deed00d61b410ef18d2e7a55f69970ccedd03767e6a753b3eae9534c40ecdb03\": rpc error: code = NotFound desc = could not find container \"deed00d61b410ef18d2e7a55f69970ccedd03767e6a753b3eae9534c40ecdb03\": container with ID starting with deed00d61b410ef18d2e7a55f69970ccedd03767e6a753b3eae9534c40ecdb03 not found: ID does not exist"
Nov 03 23:05:32.622478 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.622399    4431 scope.go:122] "RemoveContainer" containerID="f427517d211dcaf69950307e501de1630bbba1fa5221ce4cc180f1ec7e063803"
Nov 03 23:05:32.634526 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.634506    4431 scope.go:122] "RemoveContainer" containerID="f427517d211dcaf69950307e501de1630bbba1fa5221ce4cc180f1ec7e063803"
Nov 03 23:05:32.634851 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:05:32.634832    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f427517d211dcaf69950307e501de1630bbba1fa5221ce4cc180f1ec7e063803\": container with ID starting with f427517d211dcaf69950307e501de1630bbba1fa5221ce4cc180f1ec7e063803 not found: ID does not exist" containerID="f427517d211dcaf69950307e501de1630bbba1fa5221ce4cc180f1ec7e063803"
Nov 03 23:05:32.634899 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.634861    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f427517d211dcaf69950307e501de1630bbba1fa5221ce4cc180f1ec7e063803"} err="failed to get container status \"f427517d211dcaf69950307e501de1630bbba1fa5221ce4cc180f1ec7e063803\": rpc error: code = NotFound desc = could not find container \"f427517d211dcaf69950307e501de1630bbba1fa5221ce4cc180f1ec7e063803\": container with ID starting with f427517d211dcaf69950307e501de1630bbba1fa5221ce4cc180f1ec7e063803 not found: ID does not exist"
Nov 03 23:05:32.634899 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.634878    4431 scope.go:122] "RemoveContainer" containerID="3599fa82d00f100f265610a263318454b15f740dcbbbb7d563c97c5b2e3a0983"
Nov 03 23:05:32.644304 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.644282    4431 scope.go:122] "RemoveContainer" containerID="3599fa82d00f100f265610a263318454b15f740dcbbbb7d563c97c5b2e3a0983"
Nov 03 23:05:32.644605 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:05:32.644586    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3599fa82d00f100f265610a263318454b15f740dcbbbb7d563c97c5b2e3a0983\": container with ID starting with 3599fa82d00f100f265610a263318454b15f740dcbbbb7d563c97c5b2e3a0983 not found: ID does not exist" containerID="3599fa82d00f100f265610a263318454b15f740dcbbbb7d563c97c5b2e3a0983"
Nov 03 23:05:32.644669 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.644615    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3599fa82d00f100f265610a263318454b15f740dcbbbb7d563c97c5b2e3a0983"} err="failed to get container status \"3599fa82d00f100f265610a263318454b15f740dcbbbb7d563c97c5b2e3a0983\": rpc error: code = NotFound desc = could not find container \"3599fa82d00f100f265610a263318454b15f740dcbbbb7d563c97c5b2e3a0983\": container with ID starting with 3599fa82d00f100f265610a263318454b15f740dcbbbb7d563c97c5b2e3a0983 not found: ID does not exist"
Nov 03 23:05:32.700996 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.700951    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f"]
Nov 03 23:05:32.705927 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.705893    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:32.719504 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.719478    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-h5g7f"]
Nov 03 23:05:32.744701 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.744671    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj"]
Nov 03 23:05:32.770765 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.770720    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-l7fhj"]
Nov 03 23:05:32.793162 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.793130    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm"]
Nov 03 23:05:32.807821 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:32.807792    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-m27fm"]
Nov 03 23:05:33.938311 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:33.938276    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:34.087717 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:34.087684    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a04d44db-0f1c-49c5-aa5b-069a256aa783" path="/var/lib/kubelet/pods/a04d44db-0f1c-49c5-aa5b-069a256aa783/volumes"
Nov 03 23:05:34.088194 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:34.088178    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acb636f9-9d36-49e8-af20-67b01ee4c095" path="/var/lib/kubelet/pods/acb636f9-9d36-49e8-af20-67b01ee4c095/volumes"
Nov 03 23:05:34.088633 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:34.088620    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5ba197f-88fc-484c-8fbd-b52d1b601a99" path="/var/lib/kubelet/pods/e5ba197f-88fc-484c-8fbd-b52d1b601a99/volumes"
Nov 03 23:05:34.506328 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:34.506304    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9k6cb"
Nov 03 23:05:34.603599 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:34.603523    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e568c172-7301-4d7f-9499-6c3a1619c9dd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e568c172-7301-4d7f-9499-6c3a1619c9dd-istio-envoy\") pod \"e568c172-7301-4d7f-9499-6c3a1619c9dd\" (UID: \"e568c172-7301-4d7f-9499-6c3a1619c9dd\") "
Nov 03 23:05:34.603599 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:34.603592    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e568c172-7301-4d7f-9499-6c3a1619c9dd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e568c172-7301-4d7f-9499-6c3a1619c9dd-istio-podinfo\") pod \"e568c172-7301-4d7f-9499-6c3a1619c9dd\" (UID: \"e568c172-7301-4d7f-9499-6c3a1619c9dd\") "
Nov 03 23:05:34.603851 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:34.603631    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e568c172-7301-4d7f-9499-6c3a1619c9dd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e568c172-7301-4d7f-9499-6c3a1619c9dd-istio-data\") pod \"e568c172-7301-4d7f-9499-6c3a1619c9dd\" (UID: \"e568c172-7301-4d7f-9499-6c3a1619c9dd\") "
Nov 03 23:05:34.603851 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:34.603652    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e568c172-7301-4d7f-9499-6c3a1619c9dd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e568c172-7301-4d7f-9499-6c3a1619c9dd-workload-socket\") pod \"e568c172-7301-4d7f-9499-6c3a1619c9dd\" (UID: \"e568c172-7301-4d7f-9499-6c3a1619c9dd\") "
Nov 03 23:05:34.603851 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:34.603711    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e568c172-7301-4d7f-9499-6c3a1619c9dd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e568c172-7301-4d7f-9499-6c3a1619c9dd-workload-certs\") pod \"e568c172-7301-4d7f-9499-6c3a1619c9dd\" (UID: \"e568c172-7301-4d7f-9499-6c3a1619c9dd\") "
Nov 03 23:05:34.603851 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:34.603772    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e568c172-7301-4d7f-9499-6c3a1619c9dd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e568c172-7301-4d7f-9499-6c3a1619c9dd-credential-socket\") pod \"e568c172-7301-4d7f-9499-6c3a1619c9dd\" (UID: \"e568c172-7301-4d7f-9499-6c3a1619c9dd\") "
Nov 03 23:05:34.603851 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:34.603835    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e568c172-7301-4d7f-9499-6c3a1619c9dd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e568c172-7301-4d7f-9499-6c3a1619c9dd-istiod-ca-cert\") pod \"e568c172-7301-4d7f-9499-6c3a1619c9dd\" (UID: \"e568c172-7301-4d7f-9499-6c3a1619c9dd\") "
Nov 03 23:05:34.604133 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:34.603936    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e568c172-7301-4d7f-9499-6c3a1619c9dd-istio-token\" (UniqueName: \"kubernetes.io/projected/e568c172-7301-4d7f-9499-6c3a1619c9dd-istio-token\") pod \"e568c172-7301-4d7f-9499-6c3a1619c9dd\" (UID: \"e568c172-7301-4d7f-9499-6c3a1619c9dd\") "
Nov 03 23:05:34.604133 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:34.603983    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e568c172-7301-4d7f-9499-6c3a1619c9dd-kube-api-access-kwsqn\" (UniqueName: \"kubernetes.io/projected/e568c172-7301-4d7f-9499-6c3a1619c9dd-kube-api-access-kwsqn\") pod \"e568c172-7301-4d7f-9499-6c3a1619c9dd\" (UID: \"e568c172-7301-4d7f-9499-6c3a1619c9dd\") "
Nov 03 23:05:34.604133 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:34.603968    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e568c172-7301-4d7f-9499-6c3a1619c9dd-istio-data" pod "e568c172-7301-4d7f-9499-6c3a1619c9dd" (UID: "e568c172-7301-4d7f-9499-6c3a1619c9dd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:05:34.604133 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:34.604041    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e568c172-7301-4d7f-9499-6c3a1619c9dd-workload-socket" pod "e568c172-7301-4d7f-9499-6c3a1619c9dd" (UID: "e568c172-7301-4d7f-9499-6c3a1619c9dd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:05:34.604406 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:34.604201    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e568c172-7301-4d7f-9499-6c3a1619c9dd-credential-socket" pod "e568c172-7301-4d7f-9499-6c3a1619c9dd" (UID: "e568c172-7301-4d7f-9499-6c3a1619c9dd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:05:34.604406 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:34.604352    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e568c172-7301-4d7f-9499-6c3a1619c9dd-istiod-ca-cert" pod "e568c172-7301-4d7f-9499-6c3a1619c9dd" (UID: "e568c172-7301-4d7f-9499-6c3a1619c9dd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:05:34.604583 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:34.604560    4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e568c172-7301-4d7f-9499-6c3a1619c9dd-istio-data\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:05:34.604661 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:34.604591    4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e568c172-7301-4d7f-9499-6c3a1619c9dd-workload-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:05:34.604661 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:34.604607    4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e568c172-7301-4d7f-9499-6c3a1619c9dd-credential-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:05:34.604661 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:34.604622    4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e568c172-7301-4d7f-9499-6c3a1619c9dd-istiod-ca-cert\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:05:34.604828 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:34.604696    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e568c172-7301-4d7f-9499-6c3a1619c9dd-workload-certs" pod "e568c172-7301-4d7f-9499-6c3a1619c9dd" (UID: "e568c172-7301-4d7f-9499-6c3a1619c9dd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:05:34.605936 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:34.605904    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e568c172-7301-4d7f-9499-6c3a1619c9dd-istio-envoy" pod "e568c172-7301-4d7f-9499-6c3a1619c9dd" (UID: "e568c172-7301-4d7f-9499-6c3a1619c9dd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:05:34.606061 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:34.606005    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e568c172-7301-4d7f-9499-6c3a1619c9dd-istio-podinfo" pod "e568c172-7301-4d7f-9499-6c3a1619c9dd" (UID: "e568c172-7301-4d7f-9499-6c3a1619c9dd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 23:05:34.606114 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:34.606098    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e568c172-7301-4d7f-9499-6c3a1619c9dd-kube-api-access-kwsqn" pod "e568c172-7301-4d7f-9499-6c3a1619c9dd" (UID: "e568c172-7301-4d7f-9499-6c3a1619c9dd"). InnerVolumeSpecName "kube-api-access-kwsqn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:05:34.606490 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:34.606474    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e568c172-7301-4d7f-9499-6c3a1619c9dd-istio-token" pod "e568c172-7301-4d7f-9499-6c3a1619c9dd" (UID: "e568c172-7301-4d7f-9499-6c3a1619c9dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:05:34.636190 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:34.636155    4431 generic.go:356] "Generic (PLEG): container finished" podID="e568c172-7301-4d7f-9499-6c3a1619c9dd" containerID="9dcc65475817e4e163a86b4a88c69cd79305fa4f7c6b6739cb614a0aa7b4e416" exitCode=0
Nov 03 23:05:34.636349 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:34.636205    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9k6cb"
Nov 03 23:05:34.636349 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:34.636251    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9k6cb" event={"ID":"e568c172-7301-4d7f-9499-6c3a1619c9dd","Type":"ContainerDied","Data":"9dcc65475817e4e163a86b4a88c69cd79305fa4f7c6b6739cb614a0aa7b4e416"}
Nov 03 23:05:34.636349 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:34.636303    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9k6cb" event={"ID":"e568c172-7301-4d7f-9499-6c3a1619c9dd","Type":"ContainerDied","Data":"7be113d7f4e432c6a17d4b18deb2b5268782e62cec40f7ab0d8598f76d89eb33"}
Nov 03 23:05:34.636349 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:34.636327    4431 scope.go:122] "RemoveContainer" containerID="9dcc65475817e4e163a86b4a88c69cd79305fa4f7c6b6739cb614a0aa7b4e416"
Nov 03 23:05:34.646291 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:34.646273    4431 scope.go:122] "RemoveContainer" containerID="9dcc65475817e4e163a86b4a88c69cd79305fa4f7c6b6739cb614a0aa7b4e416"
Nov 03 23:05:34.646565 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:05:34.646546    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dcc65475817e4e163a86b4a88c69cd79305fa4f7c6b6739cb614a0aa7b4e416\": container with ID starting with 9dcc65475817e4e163a86b4a88c69cd79305fa4f7c6b6739cb614a0aa7b4e416 not found: ID does not exist" containerID="9dcc65475817e4e163a86b4a88c69cd79305fa4f7c6b6739cb614a0aa7b4e416"
Nov 03 23:05:34.646616 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:34.646574    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dcc65475817e4e163a86b4a88c69cd79305fa4f7c6b6739cb614a0aa7b4e416"} err="failed to get container status \"9dcc65475817e4e163a86b4a88c69cd79305fa4f7c6b6739cb614a0aa7b4e416\": rpc error: code = NotFound desc = could not find container \"9dcc65475817e4e163a86b4a88c69cd79305fa4f7c6b6739cb614a0aa7b4e416\": container with ID starting with 9dcc65475817e4e163a86b4a88c69cd79305fa4f7c6b6739cb614a0aa7b4e416 not found: ID does not exist"
Nov 03 23:05:34.687435 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:34.687408    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9k6cb"]
Nov 03 23:05:34.700383 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:34.700341    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-9k6cb"]
Nov 03 23:05:34.705413 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:34.705392    4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e568c172-7301-4d7f-9499-6c3a1619c9dd-workload-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:05:34.705413 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:34.705414    4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e568c172-7301-4d7f-9499-6c3a1619c9dd-istio-token\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:05:34.705572 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:34.705422    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwsqn\" (UniqueName: \"kubernetes.io/projected/e568c172-7301-4d7f-9499-6c3a1619c9dd-kube-api-access-kwsqn\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:05:34.705572 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:34.705431    4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e568c172-7301-4d7f-9499-6c3a1619c9dd-istio-envoy\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:05:34.705572 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:34.705448    4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e568c172-7301-4d7f-9499-6c3a1619c9dd-istio-podinfo\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:05:35.522475 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:35.522442    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:35.834863 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:35.834781    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w6-2a2eca/rate-test-model-2a2eca-kserve-7bf8dbc765-xm2vl" podUID="e4e5734b-ac12-4292-beb0-77d4aa2c08d6" containerName="main" containerID="cri-o://5cc4876f7133858a4b25b243c47556769289df93e33d0a2862f23a55b1db9847" gracePeriod=45
Nov 03 23:05:36.089345 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:36.089268    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e568c172-7301-4d7f-9499-6c3a1619c9dd" path="/var/lib/kubelet/pods/e568c172-7301-4d7f-9499-6c3a1619c9dd/volumes"
Nov 03 23:05:36.106111 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:36.106089    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-2a2eca/rate-test-model-2a2eca-kserve-7bf8dbc765-xm2vl"
Nov 03 23:05:36.222924 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:36.222887    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4e5734b-ac12-4292-beb0-77d4aa2c08d6-home\" (UniqueName: \"kubernetes.io/empty-dir/e4e5734b-ac12-4292-beb0-77d4aa2c08d6-home\") pod \"e4e5734b-ac12-4292-beb0-77d4aa2c08d6\" (UID: \"e4e5734b-ac12-4292-beb0-77d4aa2c08d6\") "
Nov 03 23:05:36.223133 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:36.222933    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4e5734b-ac12-4292-beb0-77d4aa2c08d6-kube-api-access-7p9v5\" (UniqueName: \"kubernetes.io/projected/e4e5734b-ac12-4292-beb0-77d4aa2c08d6-kube-api-access-7p9v5\") pod \"e4e5734b-ac12-4292-beb0-77d4aa2c08d6\" (UID: \"e4e5734b-ac12-4292-beb0-77d4aa2c08d6\") "
Nov 03 23:05:36.223133 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:36.222982    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4e5734b-ac12-4292-beb0-77d4aa2c08d6-tls-certs\" (UniqueName: \"kubernetes.io/secret/e4e5734b-ac12-4292-beb0-77d4aa2c08d6-tls-certs\") pod \"e4e5734b-ac12-4292-beb0-77d4aa2c08d6\" (UID: \"e4e5734b-ac12-4292-beb0-77d4aa2c08d6\") "
Nov 03 23:05:36.223133 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:36.223130    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4e5734b-ac12-4292-beb0-77d4aa2c08d6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4e5734b-ac12-4292-beb0-77d4aa2c08d6-tmp-dir\") pod \"e4e5734b-ac12-4292-beb0-77d4aa2c08d6\" (UID: \"e4e5734b-ac12-4292-beb0-77d4aa2c08d6\") "
Nov 03 23:05:36.223361 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:36.223164    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4e5734b-ac12-4292-beb0-77d4aa2c08d6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4e5734b-ac12-4292-beb0-77d4aa2c08d6-dshm\") pod \"e4e5734b-ac12-4292-beb0-77d4aa2c08d6\" (UID: \"e4e5734b-ac12-4292-beb0-77d4aa2c08d6\") "
Nov 03 23:05:36.223361 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:36.223188    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4e5734b-ac12-4292-beb0-77d4aa2c08d6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4e5734b-ac12-4292-beb0-77d4aa2c08d6-model-cache\") pod \"e4e5734b-ac12-4292-beb0-77d4aa2c08d6\" (UID: \"e4e5734b-ac12-4292-beb0-77d4aa2c08d6\") "
Nov 03 23:05:36.223361 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:36.223276    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4e5734b-ac12-4292-beb0-77d4aa2c08d6-home" pod "e4e5734b-ac12-4292-beb0-77d4aa2c08d6" (UID: "e4e5734b-ac12-4292-beb0-77d4aa2c08d6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:05:36.223573 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:36.223555    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4e5734b-ac12-4292-beb0-77d4aa2c08d6-model-cache" pod "e4e5734b-ac12-4292-beb0-77d4aa2c08d6" (UID: "e4e5734b-ac12-4292-beb0-77d4aa2c08d6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:05:36.223703 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:36.223681    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4e5734b-ac12-4292-beb0-77d4aa2c08d6-tmp-dir" pod "e4e5734b-ac12-4292-beb0-77d4aa2c08d6" (UID: "e4e5734b-ac12-4292-beb0-77d4aa2c08d6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:05:36.223750 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:36.223722    4431 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4e5734b-ac12-4292-beb0-77d4aa2c08d6-model-cache\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:05:36.223750 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:36.223735    4431 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4e5734b-ac12-4292-beb0-77d4aa2c08d6-home\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:05:36.225319 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:36.225292    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4e5734b-ac12-4292-beb0-77d4aa2c08d6-tls-certs" pod "e4e5734b-ac12-4292-beb0-77d4aa2c08d6" (UID: "e4e5734b-ac12-4292-beb0-77d4aa2c08d6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:05:36.225429 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:36.225343    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4e5734b-ac12-4292-beb0-77d4aa2c08d6-kube-api-access-7p9v5" pod "e4e5734b-ac12-4292-beb0-77d4aa2c08d6" (UID: "e4e5734b-ac12-4292-beb0-77d4aa2c08d6"). InnerVolumeSpecName "kube-api-access-7p9v5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:05:36.225477 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:36.225436    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4e5734b-ac12-4292-beb0-77d4aa2c08d6-dshm" pod "e4e5734b-ac12-4292-beb0-77d4aa2c08d6" (UID: "e4e5734b-ac12-4292-beb0-77d4aa2c08d6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:05:36.324463 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:36.324428    4431 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4e5734b-ac12-4292-beb0-77d4aa2c08d6-dshm\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:05:36.324463 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:36.324458    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7p9v5\" (UniqueName: \"kubernetes.io/projected/e4e5734b-ac12-4292-beb0-77d4aa2c08d6-kube-api-access-7p9v5\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:05:36.324463 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:36.324470    4431 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4e5734b-ac12-4292-beb0-77d4aa2c08d6-tls-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:05:36.324703 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:36.324479    4431 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4e5734b-ac12-4292-beb0-77d4aa2c08d6-tmp-dir\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:05:36.650019 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:36.649988    4431 generic.go:356] "Generic (PLEG): container finished" podID="e4e5734b-ac12-4292-beb0-77d4aa2c08d6" containerID="5cc4876f7133858a4b25b243c47556769289df93e33d0a2862f23a55b1db9847" exitCode=0
Nov 03 23:05:36.650507 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:36.650033    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-2a2eca/rate-test-model-2a2eca-kserve-7bf8dbc765-xm2vl"
Nov 03 23:05:36.650507 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:36.650070    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-2a2eca/rate-test-model-2a2eca-kserve-7bf8dbc765-xm2vl" event={"ID":"e4e5734b-ac12-4292-beb0-77d4aa2c08d6","Type":"ContainerDied","Data":"5cc4876f7133858a4b25b243c47556769289df93e33d0a2862f23a55b1db9847"}
Nov 03 23:05:36.650507 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:36.650113    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-2a2eca/rate-test-model-2a2eca-kserve-7bf8dbc765-xm2vl" event={"ID":"e4e5734b-ac12-4292-beb0-77d4aa2c08d6","Type":"ContainerDied","Data":"fc0269b4eac9ca9e19ecd8955b5e2cd241328c97f5d92cb2fdd0f16579b8f132"}
Nov 03 23:05:36.650507 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:36.650135    4431 scope.go:122] "RemoveContainer" containerID="5cc4876f7133858a4b25b243c47556769289df93e33d0a2862f23a55b1db9847"
Nov 03 23:05:36.664107 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:36.664087    4431 scope.go:122] "RemoveContainer" containerID="5cc4876f7133858a4b25b243c47556769289df93e33d0a2862f23a55b1db9847"
Nov 03 23:05:36.664439 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:05:36.664419    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cc4876f7133858a4b25b243c47556769289df93e33d0a2862f23a55b1db9847\": container with ID starting with 5cc4876f7133858a4b25b243c47556769289df93e33d0a2862f23a55b1db9847 not found: ID does not exist" containerID="5cc4876f7133858a4b25b243c47556769289df93e33d0a2862f23a55b1db9847"
Nov 03 23:05:36.664514 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:36.664451    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cc4876f7133858a4b25b243c47556769289df93e33d0a2862f23a55b1db9847"} err="failed to get container status \"5cc4876f7133858a4b25b243c47556769289df93e33d0a2862f23a55b1db9847\": rpc error: code = NotFound desc = could not find container \"5cc4876f7133858a4b25b243c47556769289df93e33d0a2862f23a55b1db9847\": container with ID starting with 5cc4876f7133858a4b25b243c47556769289df93e33d0a2862f23a55b1db9847 not found: ID does not exist"
Nov 03 23:05:36.706089 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:36.706058    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w6-2a2eca/rate-test-model-2a2eca-kserve-7bf8dbc765-xm2vl"]
Nov 03 23:05:36.717761 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:36.717733    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w6-2a2eca/rate-test-model-2a2eca-kserve-7bf8dbc765-xm2vl"]
Nov 03 23:05:37.205779 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:37.205738    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:38.087289 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:38.087241    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4e5734b-ac12-4292-beb0-77d4aa2c08d6" path="/var/lib/kubelet/pods/e4e5734b-ac12-4292-beb0-77d4aa2c08d6/volumes"
Nov 03 23:05:38.848466 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:38.848428    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:38.992047 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:38.992005    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:39.134950 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:39.133507    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9"]
Nov 03 23:05:39.134950 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:39.133770    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9" podUID="db62e992-8f6d-4cba-b260-02bf3c06e839" containerName="istio-proxy" containerID="cri-o://1fae66ed8656bee092d94a35c3c71a2bd09f1c4ecd2da89ab85dc48bec420042" gracePeriod=30
Nov 03 23:05:39.201540 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:39.201500    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:39.588356 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:39.588327    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f6dcc0-6b5944664-gzcth"
Nov 03 23:05:39.588548 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:39.588395    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f6dcc0-7fc8d968b7-4fllg"
Nov 03 23:05:40.511324 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:40.511289    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:41.284718 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:41.284680    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-2a2eca-6cf9b6f4bd-9mxw7"]
Nov 03 23:05:41.453147 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:41.453115    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:41.590594 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:41.590508    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:42.098699 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:42.098657    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:43.304013 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:43.303974    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:43.450189 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:43.450153    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:43.624940 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:43.624843    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:44.398887 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:44.398859    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9"
Nov 03 23:05:44.409929 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:44.409865    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db62e992-8f6d-4cba-b260-02bf3c06e839-kube-api-access-jmb4g\" (UniqueName: \"kubernetes.io/projected/db62e992-8f6d-4cba-b260-02bf3c06e839-kube-api-access-jmb4g\") pod \"db62e992-8f6d-4cba-b260-02bf3c06e839\" (UID: \"db62e992-8f6d-4cba-b260-02bf3c06e839\") "
Nov 03 23:05:44.410059 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:44.409934    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db62e992-8f6d-4cba-b260-02bf3c06e839-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db62e992-8f6d-4cba-b260-02bf3c06e839-workload-socket\") pod \"db62e992-8f6d-4cba-b260-02bf3c06e839\" (UID: \"db62e992-8f6d-4cba-b260-02bf3c06e839\") "
Nov 03 23:05:44.410059 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:44.409957    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/db62e992-8f6d-4cba-b260-02bf3c06e839-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db62e992-8f6d-4cba-b260-02bf3c06e839-istio-podinfo\") pod \"db62e992-8f6d-4cba-b260-02bf3c06e839\" (UID: \"db62e992-8f6d-4cba-b260-02bf3c06e839\") "
Nov 03 23:05:44.410059 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:44.410000    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db62e992-8f6d-4cba-b260-02bf3c06e839-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db62e992-8f6d-4cba-b260-02bf3c06e839-istio-envoy\") pod \"db62e992-8f6d-4cba-b260-02bf3c06e839\" (UID: \"db62e992-8f6d-4cba-b260-02bf3c06e839\") "
Nov 03 23:05:44.410059 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:44.410029    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db62e992-8f6d-4cba-b260-02bf3c06e839-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db62e992-8f6d-4cba-b260-02bf3c06e839-istio-data\") pod \"db62e992-8f6d-4cba-b260-02bf3c06e839\" (UID: \"db62e992-8f6d-4cba-b260-02bf3c06e839\") "
Nov 03 23:05:44.410313 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:44.410110    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db62e992-8f6d-4cba-b260-02bf3c06e839-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db62e992-8f6d-4cba-b260-02bf3c06e839-workload-certs\") pod \"db62e992-8f6d-4cba-b260-02bf3c06e839\" (UID: \"db62e992-8f6d-4cba-b260-02bf3c06e839\") "
Nov 03 23:05:44.410313 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:44.410161    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db62e992-8f6d-4cba-b260-02bf3c06e839-istio-token\" (UniqueName: \"kubernetes.io/projected/db62e992-8f6d-4cba-b260-02bf3c06e839-istio-token\") pod \"db62e992-8f6d-4cba-b260-02bf3c06e839\" (UID: \"db62e992-8f6d-4cba-b260-02bf3c06e839\") "
Nov 03 23:05:44.410313 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:44.410228    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db62e992-8f6d-4cba-b260-02bf3c06e839-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db62e992-8f6d-4cba-b260-02bf3c06e839-istiod-ca-cert\") pod \"db62e992-8f6d-4cba-b260-02bf3c06e839\" (UID: \"db62e992-8f6d-4cba-b260-02bf3c06e839\") "
Nov 03 23:05:44.410313 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:44.410236    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db62e992-8f6d-4cba-b260-02bf3c06e839-workload-socket" pod "db62e992-8f6d-4cba-b260-02bf3c06e839" (UID: "db62e992-8f6d-4cba-b260-02bf3c06e839"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:05:44.410313 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:44.410273    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db62e992-8f6d-4cba-b260-02bf3c06e839-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db62e992-8f6d-4cba-b260-02bf3c06e839-credential-socket\") pod \"db62e992-8f6d-4cba-b260-02bf3c06e839\" (UID: \"db62e992-8f6d-4cba-b260-02bf3c06e839\") "
Nov 03 23:05:44.410597 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:44.410387    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db62e992-8f6d-4cba-b260-02bf3c06e839-istio-data" pod "db62e992-8f6d-4cba-b260-02bf3c06e839" (UID: "db62e992-8f6d-4cba-b260-02bf3c06e839"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:05:44.410654 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:44.410638    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db62e992-8f6d-4cba-b260-02bf3c06e839-workload-certs" pod "db62e992-8f6d-4cba-b260-02bf3c06e839" (UID: "db62e992-8f6d-4cba-b260-02bf3c06e839"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:05:44.410715 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:44.410670    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db62e992-8f6d-4cba-b260-02bf3c06e839-credential-socket" pod "db62e992-8f6d-4cba-b260-02bf3c06e839" (UID: "db62e992-8f6d-4cba-b260-02bf3c06e839"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:05:44.410776 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:44.410708    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db62e992-8f6d-4cba-b260-02bf3c06e839-istiod-ca-cert" pod "db62e992-8f6d-4cba-b260-02bf3c06e839" (UID: "db62e992-8f6d-4cba-b260-02bf3c06e839"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:05:44.410834 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:44.410816    4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db62e992-8f6d-4cba-b260-02bf3c06e839-istio-data\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:05:44.410834 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:44.410830    4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db62e992-8f6d-4cba-b260-02bf3c06e839-workload-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:05:44.410943 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:44.410843    4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db62e992-8f6d-4cba-b260-02bf3c06e839-istiod-ca-cert\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:05:44.410943 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:44.410856    4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db62e992-8f6d-4cba-b260-02bf3c06e839-credential-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:05:44.410943 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:44.410869    4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db62e992-8f6d-4cba-b260-02bf3c06e839-workload-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:05:44.412191 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:44.412171    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db62e992-8f6d-4cba-b260-02bf3c06e839-istio-podinfo" pod "db62e992-8f6d-4cba-b260-02bf3c06e839" (UID: "db62e992-8f6d-4cba-b260-02bf3c06e839"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 23:05:44.412346 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:44.412331    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db62e992-8f6d-4cba-b260-02bf3c06e839-istio-token" pod "db62e992-8f6d-4cba-b260-02bf3c06e839" (UID: "db62e992-8f6d-4cba-b260-02bf3c06e839"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:05:44.412521 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:44.412501    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db62e992-8f6d-4cba-b260-02bf3c06e839-kube-api-access-jmb4g" pod "db62e992-8f6d-4cba-b260-02bf3c06e839" (UID: "db62e992-8f6d-4cba-b260-02bf3c06e839"). InnerVolumeSpecName "kube-api-access-jmb4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:05:44.412655 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:44.412639    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db62e992-8f6d-4cba-b260-02bf3c06e839-istio-envoy" pod "db62e992-8f6d-4cba-b260-02bf3c06e839" (UID: "db62e992-8f6d-4cba-b260-02bf3c06e839"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:05:44.511962 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:44.511924    4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db62e992-8f6d-4cba-b260-02bf3c06e839-istio-envoy\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:05:44.511962 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:44.511956    4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db62e992-8f6d-4cba-b260-02bf3c06e839-istio-token\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:05:44.511962 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:44.511969    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmb4g\" (UniqueName: \"kubernetes.io/projected/db62e992-8f6d-4cba-b260-02bf3c06e839-kube-api-access-jmb4g\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:05:44.512286 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:44.511981    4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db62e992-8f6d-4cba-b260-02bf3c06e839-istio-podinfo\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:05:44.703792 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:44.702594    4431 generic.go:356] "Generic (PLEG): container finished" podID="db62e992-8f6d-4cba-b260-02bf3c06e839" containerID="1fae66ed8656bee092d94a35c3c71a2bd09f1c4ecd2da89ab85dc48bec420042" exitCode=0
Nov 03 23:05:44.703792 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:44.702823    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9" event={"ID":"db62e992-8f6d-4cba-b260-02bf3c06e839","Type":"ContainerDied","Data":"1fae66ed8656bee092d94a35c3c71a2bd09f1c4ecd2da89ab85dc48bec420042"}
Nov 03 23:05:44.703792 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:44.702873    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9" event={"ID":"db62e992-8f6d-4cba-b260-02bf3c06e839","Type":"ContainerDied","Data":"f04b672d9c755d2ef75b868eb4b8ccfa3fd2842ff85502c39fdd50aa63e02207"}
Nov 03 23:05:44.703792 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:44.702903    4431 scope.go:122] "RemoveContainer" containerID="1fae66ed8656bee092d94a35c3c71a2bd09f1c4ecd2da89ab85dc48bec420042"
Nov 03 23:05:44.703792 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:44.703067    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9"
Nov 03 23:05:44.714692 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:44.714671    4431 scope.go:122] "RemoveContainer" containerID="86de258efe3b36a42c2aacc19c5e4e31ecf699a43467b762c39378d16a036b58"
Nov 03 23:05:44.723826 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:44.723805    4431 scope.go:122] "RemoveContainer" containerID="1fae66ed8656bee092d94a35c3c71a2bd09f1c4ecd2da89ab85dc48bec420042"
Nov 03 23:05:44.724122 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:05:44.724099    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fae66ed8656bee092d94a35c3c71a2bd09f1c4ecd2da89ab85dc48bec420042\": container with ID starting with 1fae66ed8656bee092d94a35c3c71a2bd09f1c4ecd2da89ab85dc48bec420042 not found: ID does not exist" containerID="1fae66ed8656bee092d94a35c3c71a2bd09f1c4ecd2da89ab85dc48bec420042"
Nov 03 23:05:44.724190 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:44.724131    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fae66ed8656bee092d94a35c3c71a2bd09f1c4ecd2da89ab85dc48bec420042"} err="failed to get container status \"1fae66ed8656bee092d94a35c3c71a2bd09f1c4ecd2da89ab85dc48bec420042\": rpc error: code = NotFound desc = could not find container \"1fae66ed8656bee092d94a35c3c71a2bd09f1c4ecd2da89ab85dc48bec420042\": container with ID starting with 1fae66ed8656bee092d94a35c3c71a2bd09f1c4ecd2da89ab85dc48bec420042 not found: ID does not exist"
Nov 03 23:05:44.724190 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:44.724152    4431 scope.go:122] "RemoveContainer" containerID="86de258efe3b36a42c2aacc19c5e4e31ecf699a43467b762c39378d16a036b58"
Nov 03 23:05:44.724414 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:05:44.724398    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86de258efe3b36a42c2aacc19c5e4e31ecf699a43467b762c39378d16a036b58\": container with ID starting with 86de258efe3b36a42c2aacc19c5e4e31ecf699a43467b762c39378d16a036b58 not found: ID does not exist" containerID="86de258efe3b36a42c2aacc19c5e4e31ecf699a43467b762c39378d16a036b58"
Nov 03 23:05:44.724476 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:44.724423    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86de258efe3b36a42c2aacc19c5e4e31ecf699a43467b762c39378d16a036b58"} err="failed to get container status \"86de258efe3b36a42c2aacc19c5e4e31ecf699a43467b762c39378d16a036b58\": rpc error: code = NotFound desc = could not find container \"86de258efe3b36a42c2aacc19c5e4e31ecf699a43467b762c39378d16a036b58\": container with ID starting with 86de258efe3b36a42c2aacc19c5e4e31ecf699a43467b762c39378d16a036b58 not found: ID does not exist"
Nov 03 23:05:44.755151 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:44.755122    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9"]
Nov 03 23:05:44.768822 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:44.768789    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-b023a2-openshift-default-7bd456f44f-wshn9"]
Nov 03 23:05:45.144046 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:45.143936    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:46.088230 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:46.088185    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db62e992-8f6d-4cba-b260-02bf3c06e839" path="/var/lib/kubelet/pods/db62e992-8f6d-4cba-b260-02bf3c06e839/volumes"
Nov 03 23:05:46.286001 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:46.285965    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-2a2eca-6cf9b6f4bd-9mxw7" podUID="c9745b64-3b13-46c2-bbd3-68730b829f6d" containerName="payload-processing" containerID="cri-o://16b7139f8bf0c6b78969042463a948e1db54a750013a7d092a6f871d4fa0bbf4" gracePeriod=25
Nov 03 23:05:46.711330 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:46.711294    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:47.990655 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:47.990615    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:48.446262 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:48.446203    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:48.594565 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:48.594533    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:48.740052 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:48.739946    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:49.046803 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:49.046707    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:50.088171 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:50.088131    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:50.265148 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:50.264942    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-9b841d-5c95644856-295pk"]
Nov 03 23:05:50.289239 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:50.287494    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f6dcc0-6b5944664-gzcth"]
Nov 03 23:05:50.289239 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:50.287762    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-f6dcc0-6b5944664-gzcth" podUID="e700154c-e286-4bb3-87e1-cd0e965cbdc4" containerName="payload-processing" containerID="cri-o://5208b62e81ac2fa2ad9ac4c6945a781a0fc472fcec0c455fb5a569e059f265bb" gracePeriod=30
Nov 03 23:05:50.301238 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:50.291071    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f6dcc0-7fc8d968b7-4fllg"]
Nov 03 23:05:50.301238 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:50.291305    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f6dcc0-7fc8d968b7-4fllg" podUID="a125bb41-5bfd-4516-b7dc-971e38e6e8a3" containerName="payload-pre-processing" containerID="cri-o://4066f46b0d6317a8d07f6ccdad3632a9f086c5c7bb72612d64c2d57ed45ddad2" gracePeriod=30
Nov 03 23:05:50.375364 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:50.375242    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:50.528256 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:50.527581    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:50.671139 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:50.670715    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f6dcc0-6b5944664-gzcth"
Nov 03 23:05:50.673449 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:50.673423    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f6dcc0-7fc8d968b7-4fllg"
Nov 03 23:05:50.741155 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:50.741109    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:50.741659 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:50.741628    4431 generic.go:356] "Generic (PLEG): container finished" podID="e700154c-e286-4bb3-87e1-cd0e965cbdc4" containerID="5208b62e81ac2fa2ad9ac4c6945a781a0fc472fcec0c455fb5a569e059f265bb" exitCode=0
Nov 03 23:05:50.741809 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:50.741673    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f6dcc0-6b5944664-gzcth"
Nov 03 23:05:50.741809 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:50.741712    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f6dcc0-6b5944664-gzcth" event={"ID":"e700154c-e286-4bb3-87e1-cd0e965cbdc4","Type":"ContainerDied","Data":"5208b62e81ac2fa2ad9ac4c6945a781a0fc472fcec0c455fb5a569e059f265bb"}
Nov 03 23:05:50.741809 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:50.741770    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f6dcc0-6b5944664-gzcth" event={"ID":"e700154c-e286-4bb3-87e1-cd0e965cbdc4","Type":"ContainerDied","Data":"f6cdc8349541e9097b65ba947cac56c02bbe01923ae1dec147586f073cf32d8e"}
Nov 03 23:05:50.741809 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:50.741801    4431 scope.go:122] "RemoveContainer" containerID="5208b62e81ac2fa2ad9ac4c6945a781a0fc472fcec0c455fb5a569e059f265bb"
Nov 03 23:05:50.743255 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:50.743235    4431 generic.go:356] "Generic (PLEG): container finished" podID="a125bb41-5bfd-4516-b7dc-971e38e6e8a3" containerID="4066f46b0d6317a8d07f6ccdad3632a9f086c5c7bb72612d64c2d57ed45ddad2" exitCode=0
Nov 03 23:05:50.743474 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:50.743441    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f6dcc0-7fc8d968b7-4fllg"
Nov 03 23:05:50.743570 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:50.743473    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f6dcc0-7fc8d968b7-4fllg" event={"ID":"a125bb41-5bfd-4516-b7dc-971e38e6e8a3","Type":"ContainerDied","Data":"4066f46b0d6317a8d07f6ccdad3632a9f086c5c7bb72612d64c2d57ed45ddad2"}
Nov 03 23:05:50.743570 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:50.743517    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f6dcc0-7fc8d968b7-4fllg" event={"ID":"a125bb41-5bfd-4516-b7dc-971e38e6e8a3","Type":"ContainerDied","Data":"593f73e7c3af3b8dd90db3e2877b9398b48e08f9d5b763dbd5ac189d3e250b72"}
Nov 03 23:05:50.752586 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:50.752556    4431 scope.go:122] "RemoveContainer" containerID="5208b62e81ac2fa2ad9ac4c6945a781a0fc472fcec0c455fb5a569e059f265bb"
Nov 03 23:05:50.752866 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:05:50.752845    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5208b62e81ac2fa2ad9ac4c6945a781a0fc472fcec0c455fb5a569e059f265bb\": container with ID starting with 5208b62e81ac2fa2ad9ac4c6945a781a0fc472fcec0c455fb5a569e059f265bb not found: ID does not exist" containerID="5208b62e81ac2fa2ad9ac4c6945a781a0fc472fcec0c455fb5a569e059f265bb"
Nov 03 23:05:50.752962 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:50.752879    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5208b62e81ac2fa2ad9ac4c6945a781a0fc472fcec0c455fb5a569e059f265bb"} err="failed to get container status \"5208b62e81ac2fa2ad9ac4c6945a781a0fc472fcec0c455fb5a569e059f265bb\": rpc error: code = NotFound desc = could not find container \"5208b62e81ac2fa2ad9ac4c6945a781a0fc472fcec0c455fb5a569e059f265bb\": container with ID starting with 5208b62e81ac2fa2ad9ac4c6945a781a0fc472fcec0c455fb5a569e059f265bb not found: ID does not exist"
Nov 03 23:05:50.752962 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:50.752903    4431 scope.go:122] "RemoveContainer" containerID="4066f46b0d6317a8d07f6ccdad3632a9f086c5c7bb72612d64c2d57ed45ddad2"
Nov 03 23:05:50.765837 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:50.765817    4431 scope.go:122] "RemoveContainer" containerID="4066f46b0d6317a8d07f6ccdad3632a9f086c5c7bb72612d64c2d57ed45ddad2"
Nov 03 23:05:50.766227 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:05:50.766179    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4066f46b0d6317a8d07f6ccdad3632a9f086c5c7bb72612d64c2d57ed45ddad2\": container with ID starting with 4066f46b0d6317a8d07f6ccdad3632a9f086c5c7bb72612d64c2d57ed45ddad2 not found: ID does not exist" containerID="4066f46b0d6317a8d07f6ccdad3632a9f086c5c7bb72612d64c2d57ed45ddad2"
Nov 03 23:05:50.766317 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:50.766258    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4066f46b0d6317a8d07f6ccdad3632a9f086c5c7bb72612d64c2d57ed45ddad2"} err="failed to get container status \"4066f46b0d6317a8d07f6ccdad3632a9f086c5c7bb72612d64c2d57ed45ddad2\": rpc error: code = NotFound desc = could not find container \"4066f46b0d6317a8d07f6ccdad3632a9f086c5c7bb72612d64c2d57ed45ddad2\": container with ID starting with 4066f46b0d6317a8d07f6ccdad3632a9f086c5c7bb72612d64c2d57ed45ddad2 not found: ID does not exist"
Nov 03 23:05:50.799805 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:50.799777    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e700154c-e286-4bb3-87e1-cd0e965cbdc4-kube-api-access-n7gp6\" (UniqueName: \"kubernetes.io/projected/e700154c-e286-4bb3-87e1-cd0e965cbdc4-kube-api-access-n7gp6\") pod \"e700154c-e286-4bb3-87e1-cd0e965cbdc4\" (UID: \"e700154c-e286-4bb3-87e1-cd0e965cbdc4\") "
Nov 03 23:05:50.799962 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:50.799863    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e700154c-e286-4bb3-87e1-cd0e965cbdc4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e700154c-e286-4bb3-87e1-cd0e965cbdc4-plugins-config-volume\") pod \"e700154c-e286-4bb3-87e1-cd0e965cbdc4\" (UID: \"e700154c-e286-4bb3-87e1-cd0e965cbdc4\") "
Nov 03 23:05:50.799962 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:50.799944    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a125bb41-5bfd-4516-b7dc-971e38e6e8a3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a125bb41-5bfd-4516-b7dc-971e38e6e8a3-plugins-config-volume\") pod \"a125bb41-5bfd-4516-b7dc-971e38e6e8a3\" (UID: \"a125bb41-5bfd-4516-b7dc-971e38e6e8a3\") "
Nov 03 23:05:50.800061 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:50.800038    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a125bb41-5bfd-4516-b7dc-971e38e6e8a3-kube-api-access-b9z5h\" (UniqueName: \"kubernetes.io/projected/a125bb41-5bfd-4516-b7dc-971e38e6e8a3-kube-api-access-b9z5h\") pod \"a125bb41-5bfd-4516-b7dc-971e38e6e8a3\" (UID: \"a125bb41-5bfd-4516-b7dc-971e38e6e8a3\") "
Nov 03 23:05:50.800320 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:50.800292    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e700154c-e286-4bb3-87e1-cd0e965cbdc4-plugins-config-volume" pod "e700154c-e286-4bb3-87e1-cd0e965cbdc4" (UID: "e700154c-e286-4bb3-87e1-cd0e965cbdc4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:05:50.800440 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:50.800332    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a125bb41-5bfd-4516-b7dc-971e38e6e8a3-plugins-config-volume" pod "a125bb41-5bfd-4516-b7dc-971e38e6e8a3" (UID: "a125bb41-5bfd-4516-b7dc-971e38e6e8a3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:05:50.800440 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:50.800436    4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e700154c-e286-4bb3-87e1-cd0e965cbdc4-plugins-config-volume\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:05:50.800523 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:50.800450    4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a125bb41-5bfd-4516-b7dc-971e38e6e8a3-plugins-config-volume\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:05:50.801951 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:50.801922    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e700154c-e286-4bb3-87e1-cd0e965cbdc4-kube-api-access-n7gp6" pod "e700154c-e286-4bb3-87e1-cd0e965cbdc4" (UID: "e700154c-e286-4bb3-87e1-cd0e965cbdc4"). InnerVolumeSpecName "kube-api-access-n7gp6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:05:50.802047 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:50.802004    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a125bb41-5bfd-4516-b7dc-971e38e6e8a3-kube-api-access-b9z5h" pod "a125bb41-5bfd-4516-b7dc-971e38e6e8a3" (UID: "a125bb41-5bfd-4516-b7dc-971e38e6e8a3"). InnerVolumeSpecName "kube-api-access-b9z5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:05:50.902048 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:50.901966    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9z5h\" (UniqueName: \"kubernetes.io/projected/a125bb41-5bfd-4516-b7dc-971e38e6e8a3-kube-api-access-b9z5h\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:05:50.902048 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:50.901996    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7gp6\" (UniqueName: \"kubernetes.io/projected/e700154c-e286-4bb3-87e1-cd0e965cbdc4-kube-api-access-n7gp6\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:05:50.993750 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:50.993713    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:51.094835 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:51.094801    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f6dcc0-7fc8d968b7-4fllg"]
Nov 03 23:05:51.102628 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:51.102594    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f6dcc0-7fc8d968b7-4fllg"]
Nov 03 23:05:51.136630 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:51.136599    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f6dcc0-6b5944664-gzcth"]
Nov 03 23:05:51.140312 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:51.140286    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f6dcc0-6b5944664-gzcth"]
Nov 03 23:05:51.146721 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:51.146691    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:51.473416 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:51.473381    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:52.087614 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:52.087581    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a125bb41-5bfd-4516-b7dc-971e38e6e8a3" path="/var/lib/kubelet/pods/a125bb41-5bfd-4516-b7dc-971e38e6e8a3/volumes"
Nov 03 23:05:52.088076 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:52.088060    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e700154c-e286-4bb3-87e1-cd0e965cbdc4" path="/var/lib/kubelet/pods/e700154c-e286-4bb3-87e1-cd0e965cbdc4/volumes"
Nov 03 23:05:52.347775 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:52.347685    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-xdjl2"]
Nov 03 23:05:52.348332 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:52.348311    4431 cpu_manager.go:414] "containerMap: removing container" podUID="a04d44db-0f1c-49c5-aa5b-069a256aa783" containerName="istio-proxy"
Nov 03 23:05:52.348332 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:52.348332    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a04d44db-0f1c-49c5-aa5b-069a256aa783" containerName="istio-proxy"
Nov 03 23:05:52.348479 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:52.348341    4431 cpu_manager.go:414] "containerMap: removing container" podUID="db62e992-8f6d-4cba-b260-02bf3c06e839" containerName="istio-proxy"
Nov 03 23:05:52.348479 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:52.348348    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db62e992-8f6d-4cba-b260-02bf3c06e839" containerName="istio-proxy"
Nov 03 23:05:52.348479 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:52.348357    4431 cpu_manager.go:414] "containerMap: removing container" podUID="e4e5734b-ac12-4292-beb0-77d4aa2c08d6" containerName="main"
Nov 03 23:05:52.348479 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:52.348363    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4e5734b-ac12-4292-beb0-77d4aa2c08d6" containerName="main"
Nov 03 23:05:52.348479 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:52.348373    4431 cpu_manager.go:414] "containerMap: removing container" podUID="a125bb41-5bfd-4516-b7dc-971e38e6e8a3" containerName="payload-pre-processing"
Nov 03 23:05:52.348479 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:52.348378    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a125bb41-5bfd-4516-b7dc-971e38e6e8a3" containerName="payload-pre-processing"
Nov 03 23:05:52.348479 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:52.348401    4431 cpu_manager.go:414] "containerMap: removing container" podUID="acb636f9-9d36-49e8-af20-67b01ee4c095" containerName="istio-proxy"
Nov 03 23:05:52.348479 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:52.348406    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acb636f9-9d36-49e8-af20-67b01ee4c095" containerName="istio-proxy"
Nov 03 23:05:52.348479 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:52.348423    4431 cpu_manager.go:414] "containerMap: removing container" podUID="e700154c-e286-4bb3-87e1-cd0e965cbdc4" containerName="payload-processing"
Nov 03 23:05:52.348479 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:52.348428    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e700154c-e286-4bb3-87e1-cd0e965cbdc4" containerName="payload-processing"
Nov 03 23:05:52.348479 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:52.348437    4431 cpu_manager.go:414] "containerMap: removing container" podUID="e5ba197f-88fc-484c-8fbd-b52d1b601a99" containerName="istio-proxy"
Nov 03 23:05:52.348479 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:52.348442    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5ba197f-88fc-484c-8fbd-b52d1b601a99" containerName="istio-proxy"
Nov 03 23:05:52.348479 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:52.348448    4431 cpu_manager.go:414] "containerMap: removing container" podUID="e568c172-7301-4d7f-9499-6c3a1619c9dd" containerName="istio-proxy"
Nov 03 23:05:52.348479 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:52.348453    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e568c172-7301-4d7f-9499-6c3a1619c9dd" containerName="istio-proxy"
Nov 03 23:05:52.349205 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:52.348532    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="acb636f9-9d36-49e8-af20-67b01ee4c095" containerName="istio-proxy"
Nov 03 23:05:52.349205 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:52.348541    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="db62e992-8f6d-4cba-b260-02bf3c06e839" containerName="istio-proxy"
Nov 03 23:05:52.349205 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:52.348548    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="e700154c-e286-4bb3-87e1-cd0e965cbdc4" containerName="payload-processing"
Nov 03 23:05:52.349205 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:52.348554    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5ba197f-88fc-484c-8fbd-b52d1b601a99" containerName="istio-proxy"
Nov 03 23:05:52.349205 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:52.348559    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="e568c172-7301-4d7f-9499-6c3a1619c9dd" containerName="istio-proxy"
Nov 03 23:05:52.349205 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:52.348566    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="a04d44db-0f1c-49c5-aa5b-069a256aa783" containerName="istio-proxy"
Nov 03 23:05:52.349205 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:52.348573    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="db62e992-8f6d-4cba-b260-02bf3c06e839" containerName="istio-proxy"
Nov 03 23:05:52.349205 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:52.348578    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4e5734b-ac12-4292-beb0-77d4aa2c08d6" containerName="main"
Nov 03 23:05:52.349205 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:52.348583    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="a125bb41-5bfd-4516-b7dc-971e38e6e8a3" containerName="payload-pre-processing"
Nov 03 23:05:52.353323 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:52.353300    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-xdjl2"
Nov 03 23:05:52.368513 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:52.368492    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-23d281-openshift-default-dockercfg-88spn\""
Nov 03 23:05:52.383666 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:52.383644    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-xdjl2"]
Nov 03 23:05:52.419586 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:52.419547    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e90b239c-aeaa-4c94-97fb-6c2fd30f9244-istio-token\") pod \"e2e-shared-a-w3-23d281-openshift-default-5cb785989-xdjl2\" (UID: \"e90b239c-aeaa-4c94-97fb-6c2fd30f9244\") " pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-xdjl2"
Nov 03 23:05:52.419801 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:52.419606    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e90b239c-aeaa-4c94-97fb-6c2fd30f9244-istiod-ca-cert\") pod \"e2e-shared-a-w3-23d281-openshift-default-5cb785989-xdjl2\" (UID: \"e90b239c-aeaa-4c94-97fb-6c2fd30f9244\") " pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-xdjl2"
Nov 03 23:05:52.419801 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:52.419753    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e90b239c-aeaa-4c94-97fb-6c2fd30f9244-istio-podinfo\") pod \"e2e-shared-a-w3-23d281-openshift-default-5cb785989-xdjl2\" (UID: \"e90b239c-aeaa-4c94-97fb-6c2fd30f9244\") " pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-xdjl2"
Nov 03 23:05:52.419801 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:52.419787    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e90b239c-aeaa-4c94-97fb-6c2fd30f9244-istio-data\") pod \"e2e-shared-a-w3-23d281-openshift-default-5cb785989-xdjl2\" (UID: \"e90b239c-aeaa-4c94-97fb-6c2fd30f9244\") " pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-xdjl2"
Nov 03 23:05:52.419934 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:52.419879    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e90b239c-aeaa-4c94-97fb-6c2fd30f9244-workload-certs\") pod \"e2e-shared-a-w3-23d281-openshift-default-5cb785989-xdjl2\" (UID: \"e90b239c-aeaa-4c94-97fb-6c2fd30f9244\") " pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-xdjl2"
Nov 03 23:05:52.419977 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:52.419934    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5zr4\" (UniqueName: \"kubernetes.io/projected/e90b239c-aeaa-4c94-97fb-6c2fd30f9244-kube-api-access-j5zr4\") pod \"e2e-shared-a-w3-23d281-openshift-default-5cb785989-xdjl2\" (UID: \"e90b239c-aeaa-4c94-97fb-6c2fd30f9244\") " pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-xdjl2"
Nov 03 23:05:52.420043 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:52.419997    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e90b239c-aeaa-4c94-97fb-6c2fd30f9244-workload-socket\") pod \"e2e-shared-a-w3-23d281-openshift-default-5cb785989-xdjl2\" (UID: \"e90b239c-aeaa-4c94-97fb-6c2fd30f9244\") " pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-xdjl2"
Nov 03 23:05:52.420126 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:52.420105    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e90b239c-aeaa-4c94-97fb-6c2fd30f9244-istio-envoy\") pod \"e2e-shared-a-w3-23d281-openshift-default-5cb785989-xdjl2\" (UID: \"e90b239c-aeaa-4c94-97fb-6c2fd30f9244\") " pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-xdjl2"
Nov 03 23:05:52.420191 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:52.420174    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e90b239c-aeaa-4c94-97fb-6c2fd30f9244-credential-socket\") pod \"e2e-shared-a-w3-23d281-openshift-default-5cb785989-xdjl2\" (UID: \"e90b239c-aeaa-4c94-97fb-6c2fd30f9244\") " pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-xdjl2"
Nov 03 23:05:52.520948 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:52.520914    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e90b239c-aeaa-4c94-97fb-6c2fd30f9244-istio-token\") pod \"e2e-shared-a-w3-23d281-openshift-default-5cb785989-xdjl2\" (UID: \"e90b239c-aeaa-4c94-97fb-6c2fd30f9244\") " pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-xdjl2"
Nov 03 23:05:52.521154 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:52.520960    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e90b239c-aeaa-4c94-97fb-6c2fd30f9244-istiod-ca-cert\") pod \"e2e-shared-a-w3-23d281-openshift-default-5cb785989-xdjl2\" (UID: \"e90b239c-aeaa-4c94-97fb-6c2fd30f9244\") " pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-xdjl2"
Nov 03 23:05:52.521154 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:52.521039    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e90b239c-aeaa-4c94-97fb-6c2fd30f9244-istio-podinfo\") pod \"e2e-shared-a-w3-23d281-openshift-default-5cb785989-xdjl2\" (UID: \"e90b239c-aeaa-4c94-97fb-6c2fd30f9244\") " pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-xdjl2"
Nov 03 23:05:52.521154 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:52.521064    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e90b239c-aeaa-4c94-97fb-6c2fd30f9244-istio-data\") pod \"e2e-shared-a-w3-23d281-openshift-default-5cb785989-xdjl2\" (UID: \"e90b239c-aeaa-4c94-97fb-6c2fd30f9244\") " pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-xdjl2"
Nov 03 23:05:52.521154 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:52.521123    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e90b239c-aeaa-4c94-97fb-6c2fd30f9244-workload-certs\") pod \"e2e-shared-a-w3-23d281-openshift-default-5cb785989-xdjl2\" (UID: \"e90b239c-aeaa-4c94-97fb-6c2fd30f9244\") " pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-xdjl2"
Nov 03 23:05:52.521396 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:52.521163    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5zr4\" (UniqueName: \"kubernetes.io/projected/e90b239c-aeaa-4c94-97fb-6c2fd30f9244-kube-api-access-j5zr4\") pod \"e2e-shared-a-w3-23d281-openshift-default-5cb785989-xdjl2\" (UID: \"e90b239c-aeaa-4c94-97fb-6c2fd30f9244\") " pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-xdjl2"
Nov 03 23:05:52.521396 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:52.521198    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e90b239c-aeaa-4c94-97fb-6c2fd30f9244-workload-socket\") pod \"e2e-shared-a-w3-23d281-openshift-default-5cb785989-xdjl2\" (UID: \"e90b239c-aeaa-4c94-97fb-6c2fd30f9244\") " pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-xdjl2"
Nov 03 23:05:52.521396 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:52.521288    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e90b239c-aeaa-4c94-97fb-6c2fd30f9244-istio-envoy\") pod \"e2e-shared-a-w3-23d281-openshift-default-5cb785989-xdjl2\" (UID: \"e90b239c-aeaa-4c94-97fb-6c2fd30f9244\") " pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-xdjl2"
Nov 03 23:05:52.521396 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:52.521327    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e90b239c-aeaa-4c94-97fb-6c2fd30f9244-credential-socket\") pod \"e2e-shared-a-w3-23d281-openshift-default-5cb785989-xdjl2\" (UID: \"e90b239c-aeaa-4c94-97fb-6c2fd30f9244\") " pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-xdjl2"
Nov 03 23:05:52.521626 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:52.521543    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e90b239c-aeaa-4c94-97fb-6c2fd30f9244-istio-data\") pod \"e2e-shared-a-w3-23d281-openshift-default-5cb785989-xdjl2\" (UID: \"e90b239c-aeaa-4c94-97fb-6c2fd30f9244\") " pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-xdjl2"
Nov 03 23:05:52.521727 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:52.521702    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e90b239c-aeaa-4c94-97fb-6c2fd30f9244-workload-certs\") pod \"e2e-shared-a-w3-23d281-openshift-default-5cb785989-xdjl2\" (UID: \"e90b239c-aeaa-4c94-97fb-6c2fd30f9244\") " pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-xdjl2"
Nov 03 23:05:52.521874 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:52.521853    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e90b239c-aeaa-4c94-97fb-6c2fd30f9244-credential-socket\") pod \"e2e-shared-a-w3-23d281-openshift-default-5cb785989-xdjl2\" (UID: \"e90b239c-aeaa-4c94-97fb-6c2fd30f9244\") " pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-xdjl2"
Nov 03 23:05:52.521949 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:52.521852    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e90b239c-aeaa-4c94-97fb-6c2fd30f9244-istiod-ca-cert\") pod \"e2e-shared-a-w3-23d281-openshift-default-5cb785989-xdjl2\" (UID: \"e90b239c-aeaa-4c94-97fb-6c2fd30f9244\") " pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-xdjl2"
Nov 03 23:05:52.522008 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:52.521957    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e90b239c-aeaa-4c94-97fb-6c2fd30f9244-workload-socket\") pod \"e2e-shared-a-w3-23d281-openshift-default-5cb785989-xdjl2\" (UID: \"e90b239c-aeaa-4c94-97fb-6c2fd30f9244\") " pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-xdjl2"
Nov 03 23:05:52.523852 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:52.523823    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e90b239c-aeaa-4c94-97fb-6c2fd30f9244-istio-envoy\") pod \"e2e-shared-a-w3-23d281-openshift-default-5cb785989-xdjl2\" (UID: \"e90b239c-aeaa-4c94-97fb-6c2fd30f9244\") " pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-xdjl2"
Nov 03 23:05:52.523979 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:52.523859    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e90b239c-aeaa-4c94-97fb-6c2fd30f9244-istio-podinfo\") pod \"e2e-shared-a-w3-23d281-openshift-default-5cb785989-xdjl2\" (UID: \"e90b239c-aeaa-4c94-97fb-6c2fd30f9244\") " pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-xdjl2"
Nov 03 23:05:52.544401 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:52.544365    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e90b239c-aeaa-4c94-97fb-6c2fd30f9244-istio-token\") pod \"e2e-shared-a-w3-23d281-openshift-default-5cb785989-xdjl2\" (UID: \"e90b239c-aeaa-4c94-97fb-6c2fd30f9244\") " pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-xdjl2"
Nov 03 23:05:52.544538 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:52.544474    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5zr4\" (UniqueName: \"kubernetes.io/projected/e90b239c-aeaa-4c94-97fb-6c2fd30f9244-kube-api-access-j5zr4\") pod \"e2e-shared-a-w3-23d281-openshift-default-5cb785989-xdjl2\" (UID: \"e90b239c-aeaa-4c94-97fb-6c2fd30f9244\") " pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-xdjl2"
Nov 03 23:05:52.667269 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:52.667204    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-xdjl2"
Nov 03 23:05:52.788676 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:52.788621    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:52.985738 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:52.985714    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-xdjl2"]
Nov 03 23:05:52.987287 ip-10-0-139-73 kubenswrapper[4431]: W1103 23:05:52.987252    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode90b239c_aeaa_4c94_97fb_6c2fd30f9244.slice/crio-052bd0431e7e1ed2878a230804870b7c31d6a9ca8a7fc7b4bf3c03fc1b159f88 WatchSource:0}: Error finding container 052bd0431e7e1ed2878a230804870b7c31d6a9ca8a7fc7b4bf3c03fc1b159f88: Status 404 returned error can't find the container with id 052bd0431e7e1ed2878a230804870b7c31d6a9ca8a7fc7b4bf3c03fc1b159f88
Nov 03 23:05:52.989731 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:52.989700    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:05:52.989795 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:52.989764    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:05:52.989835 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:52.989791    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:05:53.778679 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:53.778639    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-xdjl2" event={"ID":"e90b239c-aeaa-4c94-97fb-6c2fd30f9244","Type":"ContainerStarted","Data":"dd0275646c0e941edc9e0b4ba5c985e4156f809b470dd952bd81054bc86819c3"}
Nov 03 23:05:53.779127 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:53.778707    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-xdjl2" event={"ID":"e90b239c-aeaa-4c94-97fb-6c2fd30f9244","Type":"ContainerStarted","Data":"052bd0431e7e1ed2878a230804870b7c31d6a9ca8a7fc7b4bf3c03fc1b159f88"}
Nov 03 23:05:53.864858 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:53.864796    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-xdjl2" podStartSLOduration=1.864781273 podStartE2EDuration="1.864781273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:05:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:05:53.836455479 +0000 UTC m=+3177.862311341" watchObservedRunningTime="2026-11-03 23:05:53.864781273 +0000 UTC m=+3177.890637340"
Nov 03 23:05:54.336804 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:54.336772    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:54.482699 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:54.482657    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-2a2eca-6cf9b6f4bd-9mxw7" podUID="c9745b64-3b13-46c2-bbd3-68730b829f6d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 23:05:54.668090 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:54.668056    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-xdjl2"
Nov 03 23:05:54.669864 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:54.669832    4431 patch_prober.go:29] interesting pod/e2e-shared-a-w3-23d281-openshift-default-5cb785989-xdjl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Nov 03 23:05:54.670037 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:54.669884    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-xdjl2" podUID="e90b239c-aeaa-4c94-97fb-6c2fd30f9244" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Nov 03 23:05:55.211466 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:55.211429    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:55.266238 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:55.265443    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-9b841d-5c95644856-295pk" podUID="aa0f4c59-fbe8-4b5c-b2fb-452653ac95b9" containerName="payload-processing" containerID="cri-o://c8b1a2afbf9c7c8a81702d42f2696c5193b771830283f9b6cf505bffbb991a5f" gracePeriod=25
Nov 03 23:05:55.364509 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:55.364469    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:55.531581 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:55.531549    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:55.651240 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:55.649562    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-9b841d-5c95644856-295pk"
Nov 03 23:05:55.668495 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:55.668460    4431 patch_prober.go:29] interesting pod/e2e-shared-a-w3-23d281-openshift-default-5cb785989-xdjl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Nov 03 23:05:55.668669 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:55.668524    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-xdjl2" podUID="e90b239c-aeaa-4c94-97fb-6c2fd30f9244" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Nov 03 23:05:55.784550 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:55.784463    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa0f4c59-fbe8-4b5c-b2fb-452653ac95b9-kube-api-access-n65qs\" (UniqueName: \"kubernetes.io/projected/aa0f4c59-fbe8-4b5c-b2fb-452653ac95b9-kube-api-access-n65qs\") pod \"aa0f4c59-fbe8-4b5c-b2fb-452653ac95b9\" (UID: \"aa0f4c59-fbe8-4b5c-b2fb-452653ac95b9\") "
Nov 03 23:05:55.784550 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:55.784544    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa0f4c59-fbe8-4b5c-b2fb-452653ac95b9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa0f4c59-fbe8-4b5c-b2fb-452653ac95b9-plugins-config-volume\") pod \"aa0f4c59-fbe8-4b5c-b2fb-452653ac95b9\" (UID: \"aa0f4c59-fbe8-4b5c-b2fb-452653ac95b9\") "
Nov 03 23:05:55.784972 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:55.784946    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa0f4c59-fbe8-4b5c-b2fb-452653ac95b9-plugins-config-volume" pod "aa0f4c59-fbe8-4b5c-b2fb-452653ac95b9" (UID: "aa0f4c59-fbe8-4b5c-b2fb-452653ac95b9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:05:55.785080 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:55.785062    4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa0f4c59-fbe8-4b5c-b2fb-452653ac95b9-plugins-config-volume\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:05:55.786745 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:55.786721    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa0f4c59-fbe8-4b5c-b2fb-452653ac95b9-kube-api-access-n65qs" pod "aa0f4c59-fbe8-4b5c-b2fb-452653ac95b9" (UID: "aa0f4c59-fbe8-4b5c-b2fb-452653ac95b9"). InnerVolumeSpecName "kube-api-access-n65qs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:05:55.791866 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:55.791842    4431 generic.go:356] "Generic (PLEG): container finished" podID="aa0f4c59-fbe8-4b5c-b2fb-452653ac95b9" containerID="c8b1a2afbf9c7c8a81702d42f2696c5193b771830283f9b6cf505bffbb991a5f" exitCode=0
Nov 03 23:05:55.791989 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:55.791887    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-9b841d-5c95644856-295pk"
Nov 03 23:05:55.791989 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:55.791920    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-9b841d-5c95644856-295pk" event={"ID":"aa0f4c59-fbe8-4b5c-b2fb-452653ac95b9","Type":"ContainerDied","Data":"c8b1a2afbf9c7c8a81702d42f2696c5193b771830283f9b6cf505bffbb991a5f"}
Nov 03 23:05:55.791989 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:55.791972    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-9b841d-5c95644856-295pk" event={"ID":"aa0f4c59-fbe8-4b5c-b2fb-452653ac95b9","Type":"ContainerDied","Data":"713602736abb3dd5d720ef9f7ebd411fb543758239f1bdaf64427001b848b745"}
Nov 03 23:05:55.792189 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:55.791997    4431 scope.go:122] "RemoveContainer" containerID="c8b1a2afbf9c7c8a81702d42f2696c5193b771830283f9b6cf505bffbb991a5f"
Nov 03 23:05:55.803292 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:55.803267    4431 scope.go:122] "RemoveContainer" containerID="c8b1a2afbf9c7c8a81702d42f2696c5193b771830283f9b6cf505bffbb991a5f"
Nov 03 23:05:55.803563 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:05:55.803541    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8b1a2afbf9c7c8a81702d42f2696c5193b771830283f9b6cf505bffbb991a5f\": container with ID starting with c8b1a2afbf9c7c8a81702d42f2696c5193b771830283f9b6cf505bffbb991a5f not found: ID does not exist" containerID="c8b1a2afbf9c7c8a81702d42f2696c5193b771830283f9b6cf505bffbb991a5f"
Nov 03 23:05:55.803628 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:55.803572    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8b1a2afbf9c7c8a81702d42f2696c5193b771830283f9b6cf505bffbb991a5f"} err="failed to get container status \"c8b1a2afbf9c7c8a81702d42f2696c5193b771830283f9b6cf505bffbb991a5f\": rpc error: code = NotFound desc = could not find container \"c8b1a2afbf9c7c8a81702d42f2696c5193b771830283f9b6cf505bffbb991a5f\": container with ID starting with c8b1a2afbf9c7c8a81702d42f2696c5193b771830283f9b6cf505bffbb991a5f not found: ID does not exist"
Nov 03 23:05:55.812326 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:55.812302    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:55.864345 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:55.864300    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-9b841d-5c95644856-295pk"]
Nov 03 23:05:55.866870 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:55.866844    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-9b841d-5c95644856-295pk"]
Nov 03 23:05:55.886033 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:55.886004    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n65qs\" (UniqueName: \"kubernetes.io/projected/aa0f4c59-fbe8-4b5c-b2fb-452653ac95b9-kube-api-access-n65qs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:05:55.959242 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:55.955691    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:56.089593 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:56.089509    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa0f4c59-fbe8-4b5c-b2fb-452653ac95b9" path="/var/lib/kubelet/pods/aa0f4c59-fbe8-4b5c-b2fb-452653ac95b9/volumes"
Nov 03 23:05:56.667626 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:56.667588    4431 patch_prober.go:29] interesting pod/e2e-shared-a-w3-23d281-openshift-default-5cb785989-xdjl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Nov 03 23:05:56.668023 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:56.667639    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-xdjl2" podUID="e90b239c-aeaa-4c94-97fb-6c2fd30f9244" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Nov 03 23:05:57.437977 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:57.437935    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:57.667832 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:57.667797    4431 patch_prober.go:29] interesting pod/e2e-shared-a-w3-23d281-openshift-default-5cb785989-xdjl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Nov 03 23:05:57.668464 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:57.668428    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-xdjl2" podUID="e90b239c-aeaa-4c94-97fb-6c2fd30f9244" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Nov 03 23:05:57.781290 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:57.780876    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:57.935477 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:57.935421    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:58.085764 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:58.085679    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:58.411316 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:58.411286    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:58.668472 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:58.668383    4431 patch_prober.go:29] interesting pod/e2e-shared-a-w3-23d281-openshift-default-5cb785989-xdjl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Nov 03 23:05:58.668472 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:58.668453    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-xdjl2" podUID="e90b239c-aeaa-4c94-97fb-6c2fd30f9244" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Nov 03 23:05:58.693652 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:58.693619    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:59.176588 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:59.176537    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:59.324983 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:59.324951    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:59.616675 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:59.616591    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:05:59.717651 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:59.717614    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-xdjl2"
Nov 03 23:05:59.718290 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:59.718267    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-xdjl2"
Nov 03 23:05:59.719326 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:05:59.719305    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-xdjl2"
Nov 03 23:06:00.434897 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:00.434851    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:01.259137 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:06:01.259101    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 23:06:02.210141 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:02.210003    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:03.816328 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:03.816291    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:04.481889 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:04.481847    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-2a2eca-6cf9b6f4bd-9mxw7" podUID="c9745b64-3b13-46c2-bbd3-68730b829f6d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 23:06:05.539285 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:05.539244    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:05.878153 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:05.878069    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:06.034904 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:06.034866    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:06.220203 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:06.220168    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:06.525087 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:06.525049    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:06.548717 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:06.548682    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-2a2eca-6cf9b6f4bd-9mxw7"
Nov 03 23:06:06.610454 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:06.610424    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9745b64-3b13-46c2-bbd3-68730b829f6d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9745b64-3b13-46c2-bbd3-68730b829f6d-plugins-config-volume\") pod \"c9745b64-3b13-46c2-bbd3-68730b829f6d\" (UID: \"c9745b64-3b13-46c2-bbd3-68730b829f6d\") "
Nov 03 23:06:06.610652 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:06.610470    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9745b64-3b13-46c2-bbd3-68730b829f6d-kube-api-access-dg8tz\" (UniqueName: \"kubernetes.io/projected/c9745b64-3b13-46c2-bbd3-68730b829f6d-kube-api-access-dg8tz\") pod \"c9745b64-3b13-46c2-bbd3-68730b829f6d\" (UID: \"c9745b64-3b13-46c2-bbd3-68730b829f6d\") "
Nov 03 23:06:06.610849 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:06.610824    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9745b64-3b13-46c2-bbd3-68730b829f6d-plugins-config-volume" pod "c9745b64-3b13-46c2-bbd3-68730b829f6d" (UID: "c9745b64-3b13-46c2-bbd3-68730b829f6d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:06:06.613319 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:06.613289    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9745b64-3b13-46c2-bbd3-68730b829f6d-kube-api-access-dg8tz" pod "c9745b64-3b13-46c2-bbd3-68730b829f6d" (UID: "c9745b64-3b13-46c2-bbd3-68730b829f6d"). InnerVolumeSpecName "kube-api-access-dg8tz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:06:06.669770 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:06.669737    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:06.712015 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:06.711979    4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9745b64-3b13-46c2-bbd3-68730b829f6d-plugins-config-volume\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:06:06.712015 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:06.712007    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dg8tz\" (UniqueName: \"kubernetes.io/projected/c9745b64-3b13-46c2-bbd3-68730b829f6d-kube-api-access-dg8tz\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:06:06.833605 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:06.833502    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:06.867436 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:06.867403    4431 generic.go:356] "Generic (PLEG): container finished" podID="c9745b64-3b13-46c2-bbd3-68730b829f6d" containerID="16b7139f8bf0c6b78969042463a948e1db54a750013a7d092a6f871d4fa0bbf4" exitCode=0
Nov 03 23:06:06.867608 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:06.867462    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-2a2eca-6cf9b6f4bd-9mxw7"
Nov 03 23:06:06.867608 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:06.867487    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-2a2eca-6cf9b6f4bd-9mxw7" event={"ID":"c9745b64-3b13-46c2-bbd3-68730b829f6d","Type":"ContainerDied","Data":"16b7139f8bf0c6b78969042463a948e1db54a750013a7d092a6f871d4fa0bbf4"}
Nov 03 23:06:06.867608 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:06.867552    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-2a2eca-6cf9b6f4bd-9mxw7" event={"ID":"c9745b64-3b13-46c2-bbd3-68730b829f6d","Type":"ContainerDied","Data":"b696cbb1d9e2d6db5d6347ffb91accc92f8b71ef3886337bef2a88882b56e955"}
Nov 03 23:06:06.867608 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:06.867582    4431 scope.go:122] "RemoveContainer" containerID="16b7139f8bf0c6b78969042463a948e1db54a750013a7d092a6f871d4fa0bbf4"
Nov 03 23:06:06.878934 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:06.878911    4431 scope.go:122] "RemoveContainer" containerID="16b7139f8bf0c6b78969042463a948e1db54a750013a7d092a6f871d4fa0bbf4"
Nov 03 23:06:06.879318 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:06:06.879290    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16b7139f8bf0c6b78969042463a948e1db54a750013a7d092a6f871d4fa0bbf4\": container with ID starting with 16b7139f8bf0c6b78969042463a948e1db54a750013a7d092a6f871d4fa0bbf4 not found: ID does not exist" containerID="16b7139f8bf0c6b78969042463a948e1db54a750013a7d092a6f871d4fa0bbf4"
Nov 03 23:06:06.879378 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:06.879332    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16b7139f8bf0c6b78969042463a948e1db54a750013a7d092a6f871d4fa0bbf4"} err="failed to get container status \"16b7139f8bf0c6b78969042463a948e1db54a750013a7d092a6f871d4fa0bbf4\": rpc error: code = NotFound desc = could not find container \"16b7139f8bf0c6b78969042463a948e1db54a750013a7d092a6f871d4fa0bbf4\": container with ID starting with 16b7139f8bf0c6b78969042463a948e1db54a750013a7d092a6f871d4fa0bbf4 not found: ID does not exist"
Nov 03 23:06:06.923780 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:06.923749    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-2a2eca-6cf9b6f4bd-9mxw7"]
Nov 03 23:06:06.935120 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:06.935091    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-2a2eca-6cf9b6f4bd-9mxw7"]
Nov 03 23:06:07.004940 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:07.004906    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:07.286597 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:07.286561    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:07.441945 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:07.441916    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:08.087680 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:08.087651    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9745b64-3b13-46c2-bbd3-68730b829f6d" path="/var/lib/kubelet/pods/c9745b64-3b13-46c2-bbd3-68730b829f6d/volumes"
Nov 03 23:06:08.914135 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:08.914098    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:10.498038 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:10.497949    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:10.822491 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:10.822412    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:10.972846 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:10.972810    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:11.791165 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:11.791130    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:12.079388 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:12.079301    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:12.301815 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:12.301780    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-50d738/distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r"]
Nov 03 23:06:12.312639 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:12.312610    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-50d738/distinct-w4-50d738-kserve-668fdf8b86-5pp9p"]
Nov 03 23:06:12.322070 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:12.322041    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-50d738/premium-w4-50d738-kserve-59ccd7f995-xbfq2"]
Nov 03 23:06:12.447880 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:12.447846    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:12.610359 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:12.610329    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:12.758708 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:12.758626    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:13.360270 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:13.360199    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:14.631330 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:14.631293    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:18.853098 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:18.853062    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:18.996193 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:18.996154    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:20.232652 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:20.232614    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:20.425386 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:20.425327    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:20.572542 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:20.572440    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:20.755207 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:20.755162    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:21.164696 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:21.164660    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:22.111123 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:22.111074    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:22.263041 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:22.263006    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:23.825731 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:23.825687    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:24.990932 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:24.990899    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:25.144121 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:25.144088    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:25.233681 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:25.233646    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp"]
Nov 03 23:06:25.233944 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:25.233919    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp" podUID="c33b591b-ff1c-4385-99df-343b21eb04a8" containerName="istio-proxy" containerID="cri-o://a3361c43861b84348017c36a619439ed2938bc8bc85a25cda75748552a860184" gracePeriod=30
Nov 03 23:06:25.298372 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:25.298298    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:25.636953 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:25.636869    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:26.095645 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:26.095614    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-2d619f/distinct-w6-2d619f-kserve-74dbc56695-tnvtb"]
Nov 03 23:06:26.264024 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:26.263985    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:26.645591 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:26.645546    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:27.322961 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:27.322912    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-50d738/distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r" podUID="e45b1572-6519-4c71-884a-115eeddd86cf" containerName="main" containerID="cri-o://1eb0d38257f07af06dd086ccda66f291c38edb863e381ec546a7b5f46b1bc28a" gracePeriod=45
Nov 03 23:06:27.338756 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:27.338719    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-50d738/distinct-w4-50d738-kserve-668fdf8b86-5pp9p" podUID="ce50af62-9cb6-4379-99c7-b40630538f4f" containerName="main" containerID="cri-o://34d313d7289e50e19bdb13e70d2265b439795cfc19fa1ac6a29ecc5280fa61c3" gracePeriod=45
Nov 03 23:06:27.342796 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:27.342768    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-50d738/premium-w4-50d738-kserve-59ccd7f995-xbfq2" podUID="fa0a680b-8f0d-4b19-8217-82282764511a" containerName="main" containerID="cri-o://6463c2aea4341ac22e57b41fdde2032b0ff43ec2db6c342ed825e2a7e11cabf1" gracePeriod=45
Nov 03 23:06:27.627753 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:27.627726    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-50d738/distinct-w4-50d738-kserve-668fdf8b86-5pp9p"
Nov 03 23:06:27.630892 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:27.630870    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-50d738/distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r"
Nov 03 23:06:27.649346 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:27.649322    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-50d738/premium-w4-50d738-kserve-59ccd7f995-xbfq2"
Nov 03 23:06:27.681954 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:27.681799    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:27.757783 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:27.757749    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa0a680b-8f0d-4b19-8217-82282764511a-tls-certs\" (UniqueName: \"kubernetes.io/secret/fa0a680b-8f0d-4b19-8217-82282764511a-tls-certs\") pod \"fa0a680b-8f0d-4b19-8217-82282764511a\" (UID: \"fa0a680b-8f0d-4b19-8217-82282764511a\") "
Nov 03 23:06:27.757998 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:27.757808    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce50af62-9cb6-4379-99c7-b40630538f4f-home\" (UniqueName: \"kubernetes.io/empty-dir/ce50af62-9cb6-4379-99c7-b40630538f4f-home\") pod \"ce50af62-9cb6-4379-99c7-b40630538f4f\" (UID: \"ce50af62-9cb6-4379-99c7-b40630538f4f\") "
Nov 03 23:06:27.757998 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:27.757870    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa0a680b-8f0d-4b19-8217-82282764511a-kube-api-access-h8b5h\" (UniqueName: \"kubernetes.io/projected/fa0a680b-8f0d-4b19-8217-82282764511a-kube-api-access-h8b5h\") pod \"fa0a680b-8f0d-4b19-8217-82282764511a\" (UID: \"fa0a680b-8f0d-4b19-8217-82282764511a\") "
Nov 03 23:06:27.757998 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:27.757910    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa0a680b-8f0d-4b19-8217-82282764511a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa0a680b-8f0d-4b19-8217-82282764511a-model-cache\") pod \"fa0a680b-8f0d-4b19-8217-82282764511a\" (UID: \"fa0a680b-8f0d-4b19-8217-82282764511a\") "
Nov 03 23:06:27.757998 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:27.757947    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa0a680b-8f0d-4b19-8217-82282764511a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa0a680b-8f0d-4b19-8217-82282764511a-dshm\") pod \"fa0a680b-8f0d-4b19-8217-82282764511a\" (UID: \"fa0a680b-8f0d-4b19-8217-82282764511a\") "
Nov 03 23:06:27.757998 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:27.757978    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce50af62-9cb6-4379-99c7-b40630538f4f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce50af62-9cb6-4379-99c7-b40630538f4f-model-cache\") pod \"ce50af62-9cb6-4379-99c7-b40630538f4f\" (UID: \"ce50af62-9cb6-4379-99c7-b40630538f4f\") "
Nov 03 23:06:27.758340 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:27.758009    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa0a680b-8f0d-4b19-8217-82282764511a-home\" (UniqueName: \"kubernetes.io/empty-dir/fa0a680b-8f0d-4b19-8217-82282764511a-home\") pod \"fa0a680b-8f0d-4b19-8217-82282764511a\" (UID: \"fa0a680b-8f0d-4b19-8217-82282764511a\") "
Nov 03 23:06:27.758340 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:27.758085    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce50af62-9cb6-4379-99c7-b40630538f4f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce50af62-9cb6-4379-99c7-b40630538f4f-dshm\") pod \"ce50af62-9cb6-4379-99c7-b40630538f4f\" (UID: \"ce50af62-9cb6-4379-99c7-b40630538f4f\") "
Nov 03 23:06:27.758340 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:27.758120    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce50af62-9cb6-4379-99c7-b40630538f4f-kube-api-access-8k98d\" (UniqueName: \"kubernetes.io/projected/ce50af62-9cb6-4379-99c7-b40630538f4f-kube-api-access-8k98d\") pod \"ce50af62-9cb6-4379-99c7-b40630538f4f\" (UID: \"ce50af62-9cb6-4379-99c7-b40630538f4f\") "
Nov 03 23:06:27.758340 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:27.758125    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce50af62-9cb6-4379-99c7-b40630538f4f-home" pod "ce50af62-9cb6-4379-99c7-b40630538f4f" (UID: "ce50af62-9cb6-4379-99c7-b40630538f4f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:06:27.758340 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:27.758156    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e45b1572-6519-4c71-884a-115eeddd86cf-home\" (UniqueName: \"kubernetes.io/empty-dir/e45b1572-6519-4c71-884a-115eeddd86cf-home\") pod \"e45b1572-6519-4c71-884a-115eeddd86cf\" (UID: \"e45b1572-6519-4c71-884a-115eeddd86cf\") "
Nov 03 23:06:27.758340 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:27.758209    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e45b1572-6519-4c71-884a-115eeddd86cf-kube-api-access-428n5\" (UniqueName: \"kubernetes.io/projected/e45b1572-6519-4c71-884a-115eeddd86cf-kube-api-access-428n5\") pod \"e45b1572-6519-4c71-884a-115eeddd86cf\" (UID: \"e45b1572-6519-4c71-884a-115eeddd86cf\") "
Nov 03 23:06:27.758340 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:27.758257    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e45b1572-6519-4c71-884a-115eeddd86cf-tls-certs\" (UniqueName: \"kubernetes.io/secret/e45b1572-6519-4c71-884a-115eeddd86cf-tls-certs\") pod \"e45b1572-6519-4c71-884a-115eeddd86cf\" (UID: \"e45b1572-6519-4c71-884a-115eeddd86cf\") "
Nov 03 23:06:27.758340 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:27.758302    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e45b1572-6519-4c71-884a-115eeddd86cf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e45b1572-6519-4c71-884a-115eeddd86cf-model-cache\") pod \"e45b1572-6519-4c71-884a-115eeddd86cf\" (UID: \"e45b1572-6519-4c71-884a-115eeddd86cf\") "
Nov 03 23:06:27.758816 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:27.758347    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e45b1572-6519-4c71-884a-115eeddd86cf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e45b1572-6519-4c71-884a-115eeddd86cf-dshm\") pod \"e45b1572-6519-4c71-884a-115eeddd86cf\" (UID: \"e45b1572-6519-4c71-884a-115eeddd86cf\") "
Nov 03 23:06:27.758816 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:27.758379    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce50af62-9cb6-4379-99c7-b40630538f4f-tls-certs\" (UniqueName: \"kubernetes.io/secret/ce50af62-9cb6-4379-99c7-b40630538f4f-tls-certs\") pod \"ce50af62-9cb6-4379-99c7-b40630538f4f\" (UID: \"ce50af62-9cb6-4379-99c7-b40630538f4f\") "
Nov 03 23:06:27.758816 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:27.758450    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce50af62-9cb6-4379-99c7-b40630538f4f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce50af62-9cb6-4379-99c7-b40630538f4f-tmp-dir\") pod \"ce50af62-9cb6-4379-99c7-b40630538f4f\" (UID: \"ce50af62-9cb6-4379-99c7-b40630538f4f\") "
Nov 03 23:06:27.758816 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:27.758500    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e45b1572-6519-4c71-884a-115eeddd86cf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e45b1572-6519-4c71-884a-115eeddd86cf-tmp-dir\") pod \"e45b1572-6519-4c71-884a-115eeddd86cf\" (UID: \"e45b1572-6519-4c71-884a-115eeddd86cf\") "
Nov 03 23:06:27.758816 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:27.758531    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa0a680b-8f0d-4b19-8217-82282764511a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa0a680b-8f0d-4b19-8217-82282764511a-tmp-dir\") pod \"fa0a680b-8f0d-4b19-8217-82282764511a\" (UID: \"fa0a680b-8f0d-4b19-8217-82282764511a\") "
Nov 03 23:06:27.759102 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:27.759002    4431 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce50af62-9cb6-4379-99c7-b40630538f4f-home\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:06:27.759102 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:27.758346    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa0a680b-8f0d-4b19-8217-82282764511a-home" pod "fa0a680b-8f0d-4b19-8217-82282764511a" (UID: "fa0a680b-8f0d-4b19-8217-82282764511a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:06:27.759102 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:27.758754    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e45b1572-6519-4c71-884a-115eeddd86cf-home" pod "e45b1572-6519-4c71-884a-115eeddd86cf" (UID: "e45b1572-6519-4c71-884a-115eeddd86cf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:06:27.759102 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:27.759027    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa0a680b-8f0d-4b19-8217-82282764511a-model-cache" pod "fa0a680b-8f0d-4b19-8217-82282764511a" (UID: "fa0a680b-8f0d-4b19-8217-82282764511a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:06:27.764133 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:27.759238    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa0a680b-8f0d-4b19-8217-82282764511a-tmp-dir" pod "fa0a680b-8f0d-4b19-8217-82282764511a" (UID: "fa0a680b-8f0d-4b19-8217-82282764511a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:06:27.764133 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:27.759299    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce50af62-9cb6-4379-99c7-b40630538f4f-model-cache" pod "ce50af62-9cb6-4379-99c7-b40630538f4f" (UID: "ce50af62-9cb6-4379-99c7-b40630538f4f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:06:27.764133 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:27.759711    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e45b1572-6519-4c71-884a-115eeddd86cf-model-cache" pod "e45b1572-6519-4c71-884a-115eeddd86cf" (UID: "e45b1572-6519-4c71-884a-115eeddd86cf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:06:27.764133 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:27.760566    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa0a680b-8f0d-4b19-8217-82282764511a-tls-certs" pod "fa0a680b-8f0d-4b19-8217-82282764511a" (UID: "fa0a680b-8f0d-4b19-8217-82282764511a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:06:27.764133 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:27.760651    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa0a680b-8f0d-4b19-8217-82282764511a-kube-api-access-h8b5h" pod "fa0a680b-8f0d-4b19-8217-82282764511a" (UID: "fa0a680b-8f0d-4b19-8217-82282764511a"). InnerVolumeSpecName "kube-api-access-h8b5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:06:27.764133 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:27.760939    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e45b1572-6519-4c71-884a-115eeddd86cf-tmp-dir" pod "e45b1572-6519-4c71-884a-115eeddd86cf" (UID: "e45b1572-6519-4c71-884a-115eeddd86cf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:06:27.764133 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:27.761133    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e45b1572-6519-4c71-884a-115eeddd86cf-tls-certs" pod "e45b1572-6519-4c71-884a-115eeddd86cf" (UID: "e45b1572-6519-4c71-884a-115eeddd86cf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:06:27.764133 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:27.761531    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce50af62-9cb6-4379-99c7-b40630538f4f-tmp-dir" pod "ce50af62-9cb6-4379-99c7-b40630538f4f" (UID: "ce50af62-9cb6-4379-99c7-b40630538f4f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:06:27.764133 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:27.761880    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce50af62-9cb6-4379-99c7-b40630538f4f-tls-certs" pod "ce50af62-9cb6-4379-99c7-b40630538f4f" (UID: "ce50af62-9cb6-4379-99c7-b40630538f4f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:06:27.764133 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:27.761968    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa0a680b-8f0d-4b19-8217-82282764511a-dshm" pod "fa0a680b-8f0d-4b19-8217-82282764511a" (UID: "fa0a680b-8f0d-4b19-8217-82282764511a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:06:27.764133 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:27.762488    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e45b1572-6519-4c71-884a-115eeddd86cf-kube-api-access-428n5" pod "e45b1572-6519-4c71-884a-115eeddd86cf" (UID: "e45b1572-6519-4c71-884a-115eeddd86cf"). InnerVolumeSpecName "kube-api-access-428n5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:06:27.764133 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:27.763775    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce50af62-9cb6-4379-99c7-b40630538f4f-dshm" pod "ce50af62-9cb6-4379-99c7-b40630538f4f" (UID: "ce50af62-9cb6-4379-99c7-b40630538f4f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:06:27.764831 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:27.764274    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce50af62-9cb6-4379-99c7-b40630538f4f-kube-api-access-8k98d" pod "ce50af62-9cb6-4379-99c7-b40630538f4f" (UID: "ce50af62-9cb6-4379-99c7-b40630538f4f"). InnerVolumeSpecName "kube-api-access-8k98d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:06:27.766902 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:27.764576    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e45b1572-6519-4c71-884a-115eeddd86cf-dshm" pod "e45b1572-6519-4c71-884a-115eeddd86cf" (UID: "e45b1572-6519-4c71-884a-115eeddd86cf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:06:27.860411 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:27.860308    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-428n5\" (UniqueName: \"kubernetes.io/projected/e45b1572-6519-4c71-884a-115eeddd86cf-kube-api-access-428n5\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:06:27.860411 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:27.860349    4431 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e45b1572-6519-4c71-884a-115eeddd86cf-tls-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:06:27.860411 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:27.860366    4431 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e45b1572-6519-4c71-884a-115eeddd86cf-model-cache\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:06:27.860411 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:27.860378    4431 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e45b1572-6519-4c71-884a-115eeddd86cf-dshm\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:06:27.860411 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:27.860391    4431 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce50af62-9cb6-4379-99c7-b40630538f4f-tls-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:06:27.860411 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:27.860401    4431 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce50af62-9cb6-4379-99c7-b40630538f4f-tmp-dir\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:06:27.860411 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:27.860411    4431 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e45b1572-6519-4c71-884a-115eeddd86cf-tmp-dir\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:06:27.860893 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:27.860423    4431 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa0a680b-8f0d-4b19-8217-82282764511a-tmp-dir\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:06:27.860893 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:27.860683    4431 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa0a680b-8f0d-4b19-8217-82282764511a-tls-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:06:27.860893 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:27.860706    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8b5h\" (UniqueName: \"kubernetes.io/projected/fa0a680b-8f0d-4b19-8217-82282764511a-kube-api-access-h8b5h\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:06:27.860893 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:27.860719    4431 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa0a680b-8f0d-4b19-8217-82282764511a-model-cache\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:06:27.860893 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:27.860733    4431 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa0a680b-8f0d-4b19-8217-82282764511a-dshm\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:06:27.860893 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:27.860745    4431 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce50af62-9cb6-4379-99c7-b40630538f4f-model-cache\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:06:27.860893 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:27.860757    4431 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa0a680b-8f0d-4b19-8217-82282764511a-home\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:06:27.860893 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:27.860772    4431 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce50af62-9cb6-4379-99c7-b40630538f4f-dshm\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:06:27.860893 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:27.860784    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8k98d\" (UniqueName: \"kubernetes.io/projected/ce50af62-9cb6-4379-99c7-b40630538f4f-kube-api-access-8k98d\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:06:27.860893 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:27.860817    4431 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e45b1572-6519-4c71-884a-115eeddd86cf-home\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:06:27.985603 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:27.985564    4431 generic.go:356] "Generic (PLEG): container finished" podID="ce50af62-9cb6-4379-99c7-b40630538f4f" containerID="34d313d7289e50e19bdb13e70d2265b439795cfc19fa1ac6a29ecc5280fa61c3" exitCode=0
Nov 03 23:06:27.985778 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:27.985629    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-50d738/distinct-w4-50d738-kserve-668fdf8b86-5pp9p"
Nov 03 23:06:27.985778 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:27.985646    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-50d738/distinct-w4-50d738-kserve-668fdf8b86-5pp9p" event={"ID":"ce50af62-9cb6-4379-99c7-b40630538f4f","Type":"ContainerDied","Data":"34d313d7289e50e19bdb13e70d2265b439795cfc19fa1ac6a29ecc5280fa61c3"}
Nov 03 23:06:27.985778 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:27.985693    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-50d738/distinct-w4-50d738-kserve-668fdf8b86-5pp9p" event={"ID":"ce50af62-9cb6-4379-99c7-b40630538f4f","Type":"ContainerDied","Data":"af9703ee1a26d47150e2375316b7419e5c2f2a28add50fdd7938bdcdad9a670e"}
Nov 03 23:06:27.985778 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:27.985717    4431 scope.go:122] "RemoveContainer" containerID="34d313d7289e50e19bdb13e70d2265b439795cfc19fa1ac6a29ecc5280fa61c3"
Nov 03 23:06:27.987181 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:27.987157    4431 generic.go:356] "Generic (PLEG): container finished" podID="e45b1572-6519-4c71-884a-115eeddd86cf" containerID="1eb0d38257f07af06dd086ccda66f291c38edb863e381ec546a7b5f46b1bc28a" exitCode=0
Nov 03 23:06:27.987344 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:27.987206    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-50d738/distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r"
Nov 03 23:06:27.987344 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:27.987262    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-50d738/distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r" event={"ID":"e45b1572-6519-4c71-884a-115eeddd86cf","Type":"ContainerDied","Data":"1eb0d38257f07af06dd086ccda66f291c38edb863e381ec546a7b5f46b1bc28a"}
Nov 03 23:06:27.987344 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:27.987298    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-50d738/distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r" event={"ID":"e45b1572-6519-4c71-884a-115eeddd86cf","Type":"ContainerDied","Data":"8ec25817f8d33db3aa205f78992c12b050887ea6d446c20a233010e2dd033b80"}
Nov 03 23:06:27.989623 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:27.989597    4431 generic.go:356] "Generic (PLEG): container finished" podID="fa0a680b-8f0d-4b19-8217-82282764511a" containerID="6463c2aea4341ac22e57b41fdde2032b0ff43ec2db6c342ed825e2a7e11cabf1" exitCode=0
Nov 03 23:06:27.989743 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:27.989647    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-50d738/premium-w4-50d738-kserve-59ccd7f995-xbfq2"
Nov 03 23:06:27.989743 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:27.989648    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-50d738/premium-w4-50d738-kserve-59ccd7f995-xbfq2" event={"ID":"fa0a680b-8f0d-4b19-8217-82282764511a","Type":"ContainerDied","Data":"6463c2aea4341ac22e57b41fdde2032b0ff43ec2db6c342ed825e2a7e11cabf1"}
Nov 03 23:06:27.989743 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:27.989687    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-50d738/premium-w4-50d738-kserve-59ccd7f995-xbfq2" event={"ID":"fa0a680b-8f0d-4b19-8217-82282764511a","Type":"ContainerDied","Data":"56bb35624f160f6135b2dc0ae54751847a5eb8d14aac5ab97690df02e2a1294b"}
Nov 03 23:06:27.997900 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:27.997755    4431 scope.go:122] "RemoveContainer" containerID="34d313d7289e50e19bdb13e70d2265b439795cfc19fa1ac6a29ecc5280fa61c3"
Nov 03 23:06:27.998162 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:06:27.998078    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34d313d7289e50e19bdb13e70d2265b439795cfc19fa1ac6a29ecc5280fa61c3\": container with ID starting with 34d313d7289e50e19bdb13e70d2265b439795cfc19fa1ac6a29ecc5280fa61c3 not found: ID does not exist" containerID="34d313d7289e50e19bdb13e70d2265b439795cfc19fa1ac6a29ecc5280fa61c3"
Nov 03 23:06:27.998162 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:27.998120    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34d313d7289e50e19bdb13e70d2265b439795cfc19fa1ac6a29ecc5280fa61c3"} err="failed to get container status \"34d313d7289e50e19bdb13e70d2265b439795cfc19fa1ac6a29ecc5280fa61c3\": rpc error: code = NotFound desc = could not find container \"34d313d7289e50e19bdb13e70d2265b439795cfc19fa1ac6a29ecc5280fa61c3\": container with ID starting with 34d313d7289e50e19bdb13e70d2265b439795cfc19fa1ac6a29ecc5280fa61c3 not found: ID does not exist"
Nov 03 23:06:27.998162 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:27.998148    4431 scope.go:122] "RemoveContainer" containerID="1eb0d38257f07af06dd086ccda66f291c38edb863e381ec546a7b5f46b1bc28a"
Nov 03 23:06:28.008951 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:28.008930    4431 scope.go:122] "RemoveContainer" containerID="1eb0d38257f07af06dd086ccda66f291c38edb863e381ec546a7b5f46b1bc28a"
Nov 03 23:06:28.009263 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:06:28.009234    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1eb0d38257f07af06dd086ccda66f291c38edb863e381ec546a7b5f46b1bc28a\": container with ID starting with 1eb0d38257f07af06dd086ccda66f291c38edb863e381ec546a7b5f46b1bc28a not found: ID does not exist" containerID="1eb0d38257f07af06dd086ccda66f291c38edb863e381ec546a7b5f46b1bc28a"
Nov 03 23:06:28.009390 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:28.009267    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1eb0d38257f07af06dd086ccda66f291c38edb863e381ec546a7b5f46b1bc28a"} err="failed to get container status \"1eb0d38257f07af06dd086ccda66f291c38edb863e381ec546a7b5f46b1bc28a\": rpc error: code = NotFound desc = could not find container \"1eb0d38257f07af06dd086ccda66f291c38edb863e381ec546a7b5f46b1bc28a\": container with ID starting with 1eb0d38257f07af06dd086ccda66f291c38edb863e381ec546a7b5f46b1bc28a not found: ID does not exist"
Nov 03 23:06:28.009390 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:28.009283    4431 scope.go:122] "RemoveContainer" containerID="6463c2aea4341ac22e57b41fdde2032b0ff43ec2db6c342ed825e2a7e11cabf1"
Nov 03 23:06:28.017867 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:28.017845    4431 scope.go:122] "RemoveContainer" containerID="6463c2aea4341ac22e57b41fdde2032b0ff43ec2db6c342ed825e2a7e11cabf1"
Nov 03 23:06:28.018164 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:06:28.018144    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6463c2aea4341ac22e57b41fdde2032b0ff43ec2db6c342ed825e2a7e11cabf1\": container with ID starting with 6463c2aea4341ac22e57b41fdde2032b0ff43ec2db6c342ed825e2a7e11cabf1 not found: ID does not exist" containerID="6463c2aea4341ac22e57b41fdde2032b0ff43ec2db6c342ed825e2a7e11cabf1"
Nov 03 23:06:28.018270 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:28.018171    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6463c2aea4341ac22e57b41fdde2032b0ff43ec2db6c342ed825e2a7e11cabf1"} err="failed to get container status \"6463c2aea4341ac22e57b41fdde2032b0ff43ec2db6c342ed825e2a7e11cabf1\": rpc error: code = NotFound desc = could not find container \"6463c2aea4341ac22e57b41fdde2032b0ff43ec2db6c342ed825e2a7e11cabf1\": container with ID starting with 6463c2aea4341ac22e57b41fdde2032b0ff43ec2db6c342ed825e2a7e11cabf1 not found: ID does not exist"
Nov 03 23:06:28.067447 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:28.067419    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-50d738/premium-w4-50d738-kserve-59ccd7f995-xbfq2"]
Nov 03 23:06:28.079370 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:28.079345    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-50d738/premium-w4-50d738-kserve-59ccd7f995-xbfq2"]
Nov 03 23:06:28.087115 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:28.087086    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa0a680b-8f0d-4b19-8217-82282764511a" path="/var/lib/kubelet/pods/fa0a680b-8f0d-4b19-8217-82282764511a/volumes"
Nov 03 23:06:28.112211 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:28.112143    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-50d738/distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r"]
Nov 03 23:06:28.116356 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:28.116329    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-50d738/distinct-2-w4-50d738-kserve-f55dcc7d9-v9b6r"]
Nov 03 23:06:28.148779 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:28.148746    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-50d738/distinct-w4-50d738-kserve-668fdf8b86-5pp9p"]
Nov 03 23:06:28.152418 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:28.152392    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-50d738/distinct-w4-50d738-kserve-668fdf8b86-5pp9p"]
Nov 03 23:06:29.400532 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:29.400496    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:30.087880 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:30.087846    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce50af62-9cb6-4379-99c7-b40630538f4f" path="/var/lib/kubelet/pods/ce50af62-9cb6-4379-99c7-b40630538f4f/volumes"
Nov 03 23:06:30.088318 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:30.088303    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e45b1572-6519-4c71-884a-115eeddd86cf" path="/var/lib/kubelet/pods/e45b1572-6519-4c71-884a-115eeddd86cf/volumes"
Nov 03 23:06:30.495417 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:30.495391    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp"
Nov 03 23:06:30.695688 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:30.695653    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c33b591b-ff1c-4385-99df-343b21eb04a8-istio-token\" (UniqueName: \"kubernetes.io/projected/c33b591b-ff1c-4385-99df-343b21eb04a8-istio-token\") pod \"c33b591b-ff1c-4385-99df-343b21eb04a8\" (UID: \"c33b591b-ff1c-4385-99df-343b21eb04a8\") "
Nov 03 23:06:30.695904 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:30.695713    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c33b591b-ff1c-4385-99df-343b21eb04a8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c33b591b-ff1c-4385-99df-343b21eb04a8-istio-envoy\") pod \"c33b591b-ff1c-4385-99df-343b21eb04a8\" (UID: \"c33b591b-ff1c-4385-99df-343b21eb04a8\") "
Nov 03 23:06:30.695904 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:30.695750    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c33b591b-ff1c-4385-99df-343b21eb04a8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c33b591b-ff1c-4385-99df-343b21eb04a8-istio-podinfo\") pod \"c33b591b-ff1c-4385-99df-343b21eb04a8\" (UID: \"c33b591b-ff1c-4385-99df-343b21eb04a8\") "
Nov 03 23:06:30.695904 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:30.695844    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c33b591b-ff1c-4385-99df-343b21eb04a8-kube-api-access-s8kwk\" (UniqueName: \"kubernetes.io/projected/c33b591b-ff1c-4385-99df-343b21eb04a8-kube-api-access-s8kwk\") pod \"c33b591b-ff1c-4385-99df-343b21eb04a8\" (UID: \"c33b591b-ff1c-4385-99df-343b21eb04a8\") "
Nov 03 23:06:30.695904 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:30.695891    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c33b591b-ff1c-4385-99df-343b21eb04a8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c33b591b-ff1c-4385-99df-343b21eb04a8-workload-socket\") pod \"c33b591b-ff1c-4385-99df-343b21eb04a8\" (UID: \"c33b591b-ff1c-4385-99df-343b21eb04a8\") "
Nov 03 23:06:30.696157 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:30.695943    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c33b591b-ff1c-4385-99df-343b21eb04a8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c33b591b-ff1c-4385-99df-343b21eb04a8-credential-socket\") pod \"c33b591b-ff1c-4385-99df-343b21eb04a8\" (UID: \"c33b591b-ff1c-4385-99df-343b21eb04a8\") "
Nov 03 23:06:30.696157 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:30.695977    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c33b591b-ff1c-4385-99df-343b21eb04a8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c33b591b-ff1c-4385-99df-343b21eb04a8-istiod-ca-cert\") pod \"c33b591b-ff1c-4385-99df-343b21eb04a8\" (UID: \"c33b591b-ff1c-4385-99df-343b21eb04a8\") "
Nov 03 23:06:30.696157 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:30.696145    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c33b591b-ff1c-4385-99df-343b21eb04a8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c33b591b-ff1c-4385-99df-343b21eb04a8-istio-data\") pod \"c33b591b-ff1c-4385-99df-343b21eb04a8\" (UID: \"c33b591b-ff1c-4385-99df-343b21eb04a8\") "
Nov 03 23:06:30.696362 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:30.696203    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c33b591b-ff1c-4385-99df-343b21eb04a8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c33b591b-ff1c-4385-99df-343b21eb04a8-workload-certs\") pod \"c33b591b-ff1c-4385-99df-343b21eb04a8\" (UID: \"c33b591b-ff1c-4385-99df-343b21eb04a8\") "
Nov 03 23:06:30.696362 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:30.696204    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c33b591b-ff1c-4385-99df-343b21eb04a8-credential-socket" pod "c33b591b-ff1c-4385-99df-343b21eb04a8" (UID: "c33b591b-ff1c-4385-99df-343b21eb04a8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:06:30.696362 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:30.696249    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c33b591b-ff1c-4385-99df-343b21eb04a8-workload-socket" pod "c33b591b-ff1c-4385-99df-343b21eb04a8" (UID: "c33b591b-ff1c-4385-99df-343b21eb04a8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:06:30.696525 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:30.696446    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c33b591b-ff1c-4385-99df-343b21eb04a8-istio-data" pod "c33b591b-ff1c-4385-99df-343b21eb04a8" (UID: "c33b591b-ff1c-4385-99df-343b21eb04a8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:06:30.696587 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:30.696529    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c33b591b-ff1c-4385-99df-343b21eb04a8-workload-certs" pod "c33b591b-ff1c-4385-99df-343b21eb04a8" (UID: "c33b591b-ff1c-4385-99df-343b21eb04a8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:06:30.696587 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:30.696560    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c33b591b-ff1c-4385-99df-343b21eb04a8-istiod-ca-cert" pod "c33b591b-ff1c-4385-99df-343b21eb04a8" (UID: "c33b591b-ff1c-4385-99df-343b21eb04a8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:06:30.696787 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:30.696761    4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c33b591b-ff1c-4385-99df-343b21eb04a8-istio-data\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:06:30.696882 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:30.696785    4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c33b591b-ff1c-4385-99df-343b21eb04a8-workload-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:06:30.696882 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:30.696801    4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c33b591b-ff1c-4385-99df-343b21eb04a8-workload-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:06:30.696882 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:30.696813    4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c33b591b-ff1c-4385-99df-343b21eb04a8-credential-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:06:30.696882 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:30.696827    4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c33b591b-ff1c-4385-99df-343b21eb04a8-istiod-ca-cert\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:06:30.698391 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:30.698367    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c33b591b-ff1c-4385-99df-343b21eb04a8-istio-podinfo" pod "c33b591b-ff1c-4385-99df-343b21eb04a8" (UID: "c33b591b-ff1c-4385-99df-343b21eb04a8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 23:06:30.698512 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:30.698391    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c33b591b-ff1c-4385-99df-343b21eb04a8-istio-token" pod "c33b591b-ff1c-4385-99df-343b21eb04a8" (UID: "c33b591b-ff1c-4385-99df-343b21eb04a8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:06:30.698512 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:30.698477    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c33b591b-ff1c-4385-99df-343b21eb04a8-istio-envoy" pod "c33b591b-ff1c-4385-99df-343b21eb04a8" (UID: "c33b591b-ff1c-4385-99df-343b21eb04a8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:06:30.698512 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:30.698497    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c33b591b-ff1c-4385-99df-343b21eb04a8-kube-api-access-s8kwk" pod "c33b591b-ff1c-4385-99df-343b21eb04a8" (UID: "c33b591b-ff1c-4385-99df-343b21eb04a8"). InnerVolumeSpecName "kube-api-access-s8kwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:06:30.797806 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:30.797768    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8kwk\" (UniqueName: \"kubernetes.io/projected/c33b591b-ff1c-4385-99df-343b21eb04a8-kube-api-access-s8kwk\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:06:30.797806 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:30.797802    4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c33b591b-ff1c-4385-99df-343b21eb04a8-istio-token\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:06:30.798011 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:30.797816    4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c33b591b-ff1c-4385-99df-343b21eb04a8-istio-envoy\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:06:30.798011 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:30.797828    4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c33b591b-ff1c-4385-99df-343b21eb04a8-istio-podinfo\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:06:30.918307 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:30.918276    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:31.020134 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:31.020047    4431 generic.go:356] "Generic (PLEG): container finished" podID="c33b591b-ff1c-4385-99df-343b21eb04a8" containerID="a3361c43861b84348017c36a619439ed2938bc8bc85a25cda75748552a860184" exitCode=0
Nov 03 23:06:31.020134 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:31.020094    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp"
Nov 03 23:06:31.020134 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:31.020123    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp" event={"ID":"c33b591b-ff1c-4385-99df-343b21eb04a8","Type":"ContainerDied","Data":"a3361c43861b84348017c36a619439ed2938bc8bc85a25cda75748552a860184"}
Nov 03 23:06:31.020417 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:31.020181    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp" event={"ID":"c33b591b-ff1c-4385-99df-343b21eb04a8","Type":"ContainerDied","Data":"476b78671e9bfb93d264fbeb5ed85384c9f55717501f058085b679a5d0e6a9e4"}
Nov 03 23:06:31.020417 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:31.020229    4431 scope.go:122] "RemoveContainer" containerID="a3361c43861b84348017c36a619439ed2938bc8bc85a25cda75748552a860184"
Nov 03 23:06:31.030082 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:31.030065    4431 scope.go:122] "RemoveContainer" containerID="a3361c43861b84348017c36a619439ed2938bc8bc85a25cda75748552a860184"
Nov 03 23:06:31.030396 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:06:31.030376    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3361c43861b84348017c36a619439ed2938bc8bc85a25cda75748552a860184\": container with ID starting with a3361c43861b84348017c36a619439ed2938bc8bc85a25cda75748552a860184 not found: ID does not exist" containerID="a3361c43861b84348017c36a619439ed2938bc8bc85a25cda75748552a860184"
Nov 03 23:06:31.030479 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:31.030405    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3361c43861b84348017c36a619439ed2938bc8bc85a25cda75748552a860184"} err="failed to get container status \"a3361c43861b84348017c36a619439ed2938bc8bc85a25cda75748552a860184\": rpc error: code = NotFound desc = could not find container \"a3361c43861b84348017c36a619439ed2938bc8bc85a25cda75748552a860184\": container with ID starting with a3361c43861b84348017c36a619439ed2938bc8bc85a25cda75748552a860184 not found: ID does not exist"
Nov 03 23:06:31.082979 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:31.082948    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp"]
Nov 03 23:06:31.094662 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:31.094634    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-2kdhp"]
Nov 03 23:06:31.259119 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:06:31.259082    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 23:06:31.976806 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:31.976767    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:32.088563 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:32.088531    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c33b591b-ff1c-4385-99df-343b21eb04a8" path="/var/lib/kubelet/pods/c33b591b-ff1c-4385-99df-343b21eb04a8/volumes"
Nov 03 23:06:32.132455 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:32.132420    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:32.352550 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:32.352466    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-50d738-75496c474b-xjvd2"]
Nov 03 23:06:32.353021 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:32.352988    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-50d738-75496c474b-xjvd2" podUID="0696d31c-b588-4817-ba3b-be9d9591eef9" containerName="maas-api" containerID="cri-o://9fce25520a3806c5fce2b8c50b923ce80125e4cc0b7d2be77fe0288ce33cbb7a" gracePeriod=30
Nov 03 23:06:32.459239 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:32.449292    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:32.640241 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:32.640204    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-50d738-75496c474b-xjvd2"
Nov 03 23:06:32.720042 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:32.720003    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0696d31c-b588-4817-ba3b-be9d9591eef9-kube-api-access-mqpb8\" (UniqueName: \"kubernetes.io/projected/0696d31c-b588-4817-ba3b-be9d9591eef9-kube-api-access-mqpb8\") pod \"0696d31c-b588-4817-ba3b-be9d9591eef9\" (UID: \"0696d31c-b588-4817-ba3b-be9d9591eef9\") "
Nov 03 23:06:32.720416 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:32.720389    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0696d31c-b588-4817-ba3b-be9d9591eef9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0696d31c-b588-4817-ba3b-be9d9591eef9-metrics-certs\") pod \"0696d31c-b588-4817-ba3b-be9d9591eef9\" (UID: \"0696d31c-b588-4817-ba3b-be9d9591eef9\") "
Nov 03 23:06:32.720661 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:32.720638    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0696d31c-b588-4817-ba3b-be9d9591eef9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0696d31c-b588-4817-ba3b-be9d9591eef9-maas-api-tls\") pod \"0696d31c-b588-4817-ba3b-be9d9591eef9\" (UID: \"0696d31c-b588-4817-ba3b-be9d9591eef9\") "
Nov 03 23:06:32.721692 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:32.721644    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0696d31c-b588-4817-ba3b-be9d9591eef9-openshift-service-ca" pod "0696d31c-b588-4817-ba3b-be9d9591eef9" (UID: "0696d31c-b588-4817-ba3b-be9d9591eef9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:06:32.721843 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:32.721817    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0696d31c-b588-4817-ba3b-be9d9591eef9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0696d31c-b588-4817-ba3b-be9d9591eef9-openshift-service-ca\") pod \"0696d31c-b588-4817-ba3b-be9d9591eef9\" (UID: \"0696d31c-b588-4817-ba3b-be9d9591eef9\") "
Nov 03 23:06:32.722248 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:32.722208    4431 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0696d31c-b588-4817-ba3b-be9d9591eef9-openshift-service-ca\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:06:32.723169 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:32.722935    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0696d31c-b588-4817-ba3b-be9d9591eef9-kube-api-access-mqpb8" pod "0696d31c-b588-4817-ba3b-be9d9591eef9" (UID: "0696d31c-b588-4817-ba3b-be9d9591eef9"). InnerVolumeSpecName "kube-api-access-mqpb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:06:32.728345 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:32.723610    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0696d31c-b588-4817-ba3b-be9d9591eef9-metrics-certs" pod "0696d31c-b588-4817-ba3b-be9d9591eef9" (UID: "0696d31c-b588-4817-ba3b-be9d9591eef9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:06:32.728345 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:32.724371    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0696d31c-b588-4817-ba3b-be9d9591eef9-maas-api-tls" pod "0696d31c-b588-4817-ba3b-be9d9591eef9" (UID: "0696d31c-b588-4817-ba3b-be9d9591eef9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:06:32.823965 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:32.823924    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqpb8\" (UniqueName: \"kubernetes.io/projected/0696d31c-b588-4817-ba3b-be9d9591eef9-kube-api-access-mqpb8\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:06:32.823965 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:32.823966    4431 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0696d31c-b588-4817-ba3b-be9d9591eef9-metrics-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:06:32.824196 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:32.823982    4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0696d31c-b588-4817-ba3b-be9d9591eef9-maas-api-tls\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:06:32.975126 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:32.975092    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:33.035047 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:33.035014    4431 generic.go:356] "Generic (PLEG): container finished" podID="0696d31c-b588-4817-ba3b-be9d9591eef9" containerID="9fce25520a3806c5fce2b8c50b923ce80125e4cc0b7d2be77fe0288ce33cbb7a" exitCode=0
Nov 03 23:06:33.035513 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:33.035082    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-50d738-75496c474b-xjvd2"
Nov 03 23:06:33.035513 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:33.035102    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-50d738-75496c474b-xjvd2" event={"ID":"0696d31c-b588-4817-ba3b-be9d9591eef9","Type":"ContainerDied","Data":"9fce25520a3806c5fce2b8c50b923ce80125e4cc0b7d2be77fe0288ce33cbb7a"}
Nov 03 23:06:33.035513 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:33.035159    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-50d738-75496c474b-xjvd2" event={"ID":"0696d31c-b588-4817-ba3b-be9d9591eef9","Type":"ContainerDied","Data":"fdf31988aefc04513129196be1db4ae3ecf3dd2ce5306587a84e3d1acaec8f12"}
Nov 03 23:06:33.035513 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:33.035185    4431 scope.go:122] "RemoveContainer" containerID="9fce25520a3806c5fce2b8c50b923ce80125e4cc0b7d2be77fe0288ce33cbb7a"
Nov 03 23:06:33.045100 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:33.045079    4431 scope.go:122] "RemoveContainer" containerID="9fce25520a3806c5fce2b8c50b923ce80125e4cc0b7d2be77fe0288ce33cbb7a"
Nov 03 23:06:33.045415 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:06:33.045394    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fce25520a3806c5fce2b8c50b923ce80125e4cc0b7d2be77fe0288ce33cbb7a\": container with ID starting with 9fce25520a3806c5fce2b8c50b923ce80125e4cc0b7d2be77fe0288ce33cbb7a not found: ID does not exist" containerID="9fce25520a3806c5fce2b8c50b923ce80125e4cc0b7d2be77fe0288ce33cbb7a"
Nov 03 23:06:33.045523 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:33.045421    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fce25520a3806c5fce2b8c50b923ce80125e4cc0b7d2be77fe0288ce33cbb7a"} err="failed to get container status \"9fce25520a3806c5fce2b8c50b923ce80125e4cc0b7d2be77fe0288ce33cbb7a\": rpc error: code = NotFound desc = could not find container \"9fce25520a3806c5fce2b8c50b923ce80125e4cc0b7d2be77fe0288ce33cbb7a\": container with ID starting with 9fce25520a3806c5fce2b8c50b923ce80125e4cc0b7d2be77fe0288ce33cbb7a not found: ID does not exist"
Nov 03 23:06:33.085828 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:33.085760    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-50d738-75496c474b-xjvd2"]
Nov 03 23:06:33.097812 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:33.097787    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-50d738-75496c474b-xjvd2"]
Nov 03 23:06:33.129371 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:33.129333    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:33.577791 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:33.577754    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:34.086783 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:34.086746    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0696d31c-b588-4817-ba3b-be9d9591eef9" path="/var/lib/kubelet/pods/0696d31c-b588-4817-ba3b-be9d9591eef9/volumes"
Nov 03 23:06:34.986913 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:34.986880    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:36.288907 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:36.288865    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-434ce3-f9c44bc7b-mb2rs"]
Nov 03 23:06:36.289481 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:36.289153    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-434ce3-f9c44bc7b-mb2rs" podUID="0f78011a-db80-4c8f-b562-a80b6970d207" containerName="maas-api" containerID="cri-o://5bb4b0ecdae28fbccd17193cfbeadcd3ece64a89bd7dec712168772a0a12453a" gracePeriod=30
Nov 03 23:06:36.394400 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:36.394359    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:36.542936 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:36.542857    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:36.547940 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:36.547915    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-434ce3-f9c44bc7b-mb2rs"
Nov 03 23:06:36.666777 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:36.666732    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f78011a-db80-4c8f-b562-a80b6970d207-kube-api-access-svpzs\" (UniqueName: \"kubernetes.io/projected/0f78011a-db80-4c8f-b562-a80b6970d207-kube-api-access-svpzs\") pod \"0f78011a-db80-4c8f-b562-a80b6970d207\" (UID: \"0f78011a-db80-4c8f-b562-a80b6970d207\") "
Nov 03 23:06:36.666987 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:36.666800    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f78011a-db80-4c8f-b562-a80b6970d207-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f78011a-db80-4c8f-b562-a80b6970d207-maas-api-tls\") pod \"0f78011a-db80-4c8f-b562-a80b6970d207\" (UID: \"0f78011a-db80-4c8f-b562-a80b6970d207\") "
Nov 03 23:06:36.666987 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:36.666837    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f78011a-db80-4c8f-b562-a80b6970d207-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f78011a-db80-4c8f-b562-a80b6970d207-openshift-service-ca\") pod \"0f78011a-db80-4c8f-b562-a80b6970d207\" (UID: \"0f78011a-db80-4c8f-b562-a80b6970d207\") "
Nov 03 23:06:36.666987 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:36.666870    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f78011a-db80-4c8f-b562-a80b6970d207-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f78011a-db80-4c8f-b562-a80b6970d207-metrics-certs\") pod \"0f78011a-db80-4c8f-b562-a80b6970d207\" (UID: \"0f78011a-db80-4c8f-b562-a80b6970d207\") "
Nov 03 23:06:36.667283 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:36.667255    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f78011a-db80-4c8f-b562-a80b6970d207-openshift-service-ca" pod "0f78011a-db80-4c8f-b562-a80b6970d207" (UID: "0f78011a-db80-4c8f-b562-a80b6970d207"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:06:36.667408 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:36.667356    4431 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f78011a-db80-4c8f-b562-a80b6970d207-openshift-service-ca\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:06:36.669036 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:36.669013    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f78011a-db80-4c8f-b562-a80b6970d207-kube-api-access-svpzs" pod "0f78011a-db80-4c8f-b562-a80b6970d207" (UID: "0f78011a-db80-4c8f-b562-a80b6970d207"). InnerVolumeSpecName "kube-api-access-svpzs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:06:36.669388 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:36.669369    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f78011a-db80-4c8f-b562-a80b6970d207-maas-api-tls" pod "0f78011a-db80-4c8f-b562-a80b6970d207" (UID: "0f78011a-db80-4c8f-b562-a80b6970d207"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:06:36.669479 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:36.669396    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f78011a-db80-4c8f-b562-a80b6970d207-metrics-certs" pod "0f78011a-db80-4c8f-b562-a80b6970d207" (UID: "0f78011a-db80-4c8f-b562-a80b6970d207"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:06:36.768161 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:36.768128    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svpzs\" (UniqueName: \"kubernetes.io/projected/0f78011a-db80-4c8f-b562-a80b6970d207-kube-api-access-svpzs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:06:36.768161 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:36.768157    4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f78011a-db80-4c8f-b562-a80b6970d207-maas-api-tls\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:06:36.768161 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:36.768167    4431 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f78011a-db80-4c8f-b562-a80b6970d207-metrics-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:06:36.869183 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:36.869081    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:37.029691 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:37.029663    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:37.059584 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:37.059539    4431 generic.go:356] "Generic (PLEG): container finished" podID="0f78011a-db80-4c8f-b562-a80b6970d207" containerID="5bb4b0ecdae28fbccd17193cfbeadcd3ece64a89bd7dec712168772a0a12453a" exitCode=0
Nov 03 23:06:37.059797 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:37.059605    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-434ce3-f9c44bc7b-mb2rs"
Nov 03 23:06:37.059797 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:37.059618    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-434ce3-f9c44bc7b-mb2rs" event={"ID":"0f78011a-db80-4c8f-b562-a80b6970d207","Type":"ContainerDied","Data":"5bb4b0ecdae28fbccd17193cfbeadcd3ece64a89bd7dec712168772a0a12453a"}
Nov 03 23:06:37.059797 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:37.059686    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-434ce3-f9c44bc7b-mb2rs" event={"ID":"0f78011a-db80-4c8f-b562-a80b6970d207","Type":"ContainerDied","Data":"2eacc6da03ccd38d2ff738cf62e3980f4c603bdaf81b68f8e5254c7d43a70bd3"}
Nov 03 23:06:37.059797 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:37.059713    4431 scope.go:122] "RemoveContainer" containerID="5bb4b0ecdae28fbccd17193cfbeadcd3ece64a89bd7dec712168772a0a12453a"
Nov 03 23:06:37.073348 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:37.073326    4431 scope.go:122] "RemoveContainer" containerID="5bb4b0ecdae28fbccd17193cfbeadcd3ece64a89bd7dec712168772a0a12453a"
Nov 03 23:06:37.073682 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:06:37.073658    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bb4b0ecdae28fbccd17193cfbeadcd3ece64a89bd7dec712168772a0a12453a\": container with ID starting with 5bb4b0ecdae28fbccd17193cfbeadcd3ece64a89bd7dec712168772a0a12453a not found: ID does not exist" containerID="5bb4b0ecdae28fbccd17193cfbeadcd3ece64a89bd7dec712168772a0a12453a"
Nov 03 23:06:37.073750 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:37.073696    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bb4b0ecdae28fbccd17193cfbeadcd3ece64a89bd7dec712168772a0a12453a"} err="failed to get container status \"5bb4b0ecdae28fbccd17193cfbeadcd3ece64a89bd7dec712168772a0a12453a\": rpc error: code = NotFound desc = could not find container \"5bb4b0ecdae28fbccd17193cfbeadcd3ece64a89bd7dec712168772a0a12453a\": container with ID starting with 5bb4b0ecdae28fbccd17193cfbeadcd3ece64a89bd7dec712168772a0a12453a not found: ID does not exist"
Nov 03 23:06:37.110607 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:37.110577    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-434ce3-f9c44bc7b-mb2rs"]
Nov 03 23:06:37.122467 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:37.122392    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-434ce3-f9c44bc7b-mb2rs"]
Nov 03 23:06:37.178282 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:37.178249    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:37.480442 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:37.480406    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-23d281-97b8bbd66-7dgnl"]
Nov 03 23:06:37.534605 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:37.534574    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-23d281-8599fb6555-qtrjc"]
Nov 03 23:06:37.576765 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:37.576732    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:37.745697 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:37.745620    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:38.089942 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:38.089846    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f78011a-db80-4c8f-b562-a80b6970d207" path="/var/lib/kubelet/pods/0f78011a-db80-4c8f-b562-a80b6970d207/volumes"
Nov 03 23:06:38.904278 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:38.904244    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:40.259096 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:40.259051    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:40.703423 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:40.703382    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:40.847901 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:40.847861    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:41.102712 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:41.102629    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:41.123791 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:41.123690    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-2d619f/distinct-w6-2d619f-kserve-74dbc56695-tnvtb" podUID="90310d72-c818-4fdd-b4e7-b51f633caa05" containerName="main" containerID="cri-o://9dd05235d00df0d2082578e750a3ee62af070edd7e416545b7ef7d4f2ec94c6d" gracePeriod=45
Nov 03 23:06:41.354284 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:41.347039    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:41.455327 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:41.455275    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-2d619f/distinct-w6-2d619f-kserve-74dbc56695-tnvtb"
Nov 03 23:06:41.516114 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:41.516084    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:41.523158 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:41.523131    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90310d72-c818-4fdd-b4e7-b51f633caa05-dshm\" (UniqueName: \"kubernetes.io/empty-dir/90310d72-c818-4fdd-b4e7-b51f633caa05-dshm\") pod \"90310d72-c818-4fdd-b4e7-b51f633caa05\" (UID: \"90310d72-c818-4fdd-b4e7-b51f633caa05\") "
Nov 03 23:06:41.523335 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:41.523204    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90310d72-c818-4fdd-b4e7-b51f633caa05-tls-certs\" (UniqueName: \"kubernetes.io/secret/90310d72-c818-4fdd-b4e7-b51f633caa05-tls-certs\") pod \"90310d72-c818-4fdd-b4e7-b51f633caa05\" (UID: \"90310d72-c818-4fdd-b4e7-b51f633caa05\") "
Nov 03 23:06:41.523335 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:41.523289    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90310d72-c818-4fdd-b4e7-b51f633caa05-kube-api-access-pvsks\" (UniqueName: \"kubernetes.io/projected/90310d72-c818-4fdd-b4e7-b51f633caa05-kube-api-access-pvsks\") pod \"90310d72-c818-4fdd-b4e7-b51f633caa05\" (UID: \"90310d72-c818-4fdd-b4e7-b51f633caa05\") "
Nov 03 23:06:41.523465 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:41.523344    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90310d72-c818-4fdd-b4e7-b51f633caa05-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90310d72-c818-4fdd-b4e7-b51f633caa05-model-cache\") pod \"90310d72-c818-4fdd-b4e7-b51f633caa05\" (UID: \"90310d72-c818-4fdd-b4e7-b51f633caa05\") "
Nov 03 23:06:41.523465 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:41.523429    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90310d72-c818-4fdd-b4e7-b51f633caa05-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90310d72-c818-4fdd-b4e7-b51f633caa05-tmp-dir\") pod \"90310d72-c818-4fdd-b4e7-b51f633caa05\" (UID: \"90310d72-c818-4fdd-b4e7-b51f633caa05\") "
Nov 03 23:06:41.523568 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:41.523480    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90310d72-c818-4fdd-b4e7-b51f633caa05-home\" (UniqueName: \"kubernetes.io/empty-dir/90310d72-c818-4fdd-b4e7-b51f633caa05-home\") pod \"90310d72-c818-4fdd-b4e7-b51f633caa05\" (UID: \"90310d72-c818-4fdd-b4e7-b51f633caa05\") "
Nov 03 23:06:41.523685 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:41.523655    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90310d72-c818-4fdd-b4e7-b51f633caa05-model-cache" pod "90310d72-c818-4fdd-b4e7-b51f633caa05" (UID: "90310d72-c818-4fdd-b4e7-b51f633caa05"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:06:41.523806 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:41.523776    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90310d72-c818-4fdd-b4e7-b51f633caa05-tmp-dir" pod "90310d72-c818-4fdd-b4e7-b51f633caa05" (UID: "90310d72-c818-4fdd-b4e7-b51f633caa05"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:06:41.523892 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:41.523868    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90310d72-c818-4fdd-b4e7-b51f633caa05-home" pod "90310d72-c818-4fdd-b4e7-b51f633caa05" (UID: "90310d72-c818-4fdd-b4e7-b51f633caa05"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:06:41.523956 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:41.523943    4431 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90310d72-c818-4fdd-b4e7-b51f633caa05-model-cache\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:06:41.524023 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:41.523965    4431 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90310d72-c818-4fdd-b4e7-b51f633caa05-tmp-dir\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:06:41.525964 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:41.525941    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90310d72-c818-4fdd-b4e7-b51f633caa05-tls-certs" pod "90310d72-c818-4fdd-b4e7-b51f633caa05" (UID: "90310d72-c818-4fdd-b4e7-b51f633caa05"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:06:41.526154 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:41.526135    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90310d72-c818-4fdd-b4e7-b51f633caa05-kube-api-access-pvsks" pod "90310d72-c818-4fdd-b4e7-b51f633caa05" (UID: "90310d72-c818-4fdd-b4e7-b51f633caa05"). InnerVolumeSpecName "kube-api-access-pvsks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:06:41.526154 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:41.526140    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90310d72-c818-4fdd-b4e7-b51f633caa05-dshm" pod "90310d72-c818-4fdd-b4e7-b51f633caa05" (UID: "90310d72-c818-4fdd-b4e7-b51f633caa05"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:06:41.625284 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:41.625159    4431 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90310d72-c818-4fdd-b4e7-b51f633caa05-dshm\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:06:41.625284 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:41.625197    4431 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90310d72-c818-4fdd-b4e7-b51f633caa05-tls-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:06:41.625284 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:41.625229    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvsks\" (UniqueName: \"kubernetes.io/projected/90310d72-c818-4fdd-b4e7-b51f633caa05-kube-api-access-pvsks\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:06:41.625284 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:41.625243    4431 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90310d72-c818-4fdd-b4e7-b51f633caa05-home\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:06:42.087489 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:42.087445    4431 generic.go:356] "Generic (PLEG): container finished" podID="90310d72-c818-4fdd-b4e7-b51f633caa05" containerID="9dd05235d00df0d2082578e750a3ee62af070edd7e416545b7ef7d4f2ec94c6d" exitCode=0
Nov 03 23:06:42.087656 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:42.087548    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-2d619f/distinct-w6-2d619f-kserve-74dbc56695-tnvtb"
Nov 03 23:06:42.087979 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:42.087945    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-2d619f/distinct-w6-2d619f-kserve-74dbc56695-tnvtb" event={"ID":"90310d72-c818-4fdd-b4e7-b51f633caa05","Type":"ContainerDied","Data":"9dd05235d00df0d2082578e750a3ee62af070edd7e416545b7ef7d4f2ec94c6d"}
Nov 03 23:06:42.088108 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:42.087988    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-2d619f/distinct-w6-2d619f-kserve-74dbc56695-tnvtb" event={"ID":"90310d72-c818-4fdd-b4e7-b51f633caa05","Type":"ContainerDied","Data":"c11b90afa48515845b5bef4382d60f8d16058e528e9de82e51ec2d5d31a3dbc9"}
Nov 03 23:06:42.088108 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:42.088021    4431 scope.go:122] "RemoveContainer" containerID="9dd05235d00df0d2082578e750a3ee62af070edd7e416545b7ef7d4f2ec94c6d"
Nov 03 23:06:42.098469 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:42.098451    4431 scope.go:122] "RemoveContainer" containerID="9dd05235d00df0d2082578e750a3ee62af070edd7e416545b7ef7d4f2ec94c6d"
Nov 03 23:06:42.098785 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:06:42.098762    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dd05235d00df0d2082578e750a3ee62af070edd7e416545b7ef7d4f2ec94c6d\": container with ID starting with 9dd05235d00df0d2082578e750a3ee62af070edd7e416545b7ef7d4f2ec94c6d not found: ID does not exist" containerID="9dd05235d00df0d2082578e750a3ee62af070edd7e416545b7ef7d4f2ec94c6d"
Nov 03 23:06:42.098886 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:42.098793    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dd05235d00df0d2082578e750a3ee62af070edd7e416545b7ef7d4f2ec94c6d"} err="failed to get container status \"9dd05235d00df0d2082578e750a3ee62af070edd7e416545b7ef7d4f2ec94c6d\": rpc error: code = NotFound desc = could not find container \"9dd05235d00df0d2082578e750a3ee62af070edd7e416545b7ef7d4f2ec94c6d\": container with ID starting with 9dd05235d00df0d2082578e750a3ee62af070edd7e416545b7ef7d4f2ec94c6d not found: ID does not exist"
Nov 03 23:06:42.138099 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:42.138065    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-2d619f/distinct-w6-2d619f-kserve-74dbc56695-tnvtb"]
Nov 03 23:06:42.149422 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:42.149394    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-2d619f/distinct-w6-2d619f-kserve-74dbc56695-tnvtb"]
Nov 03 23:06:42.481083 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:42.481028    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-23d281-97b8bbd66-7dgnl" podUID="2452e573-42a1-40b4-b86f-3678286375bc" containerName="payload-processing" containerID="cri-o://936af0258adcab8270e0bc34efc0e1efc8bbc52867a87acbb0672153ba8166b5" gracePeriod=25
Nov 03 23:06:42.535856 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:42.535814    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-23d281-8599fb6555-qtrjc" podUID="98293f18-47d0-49fe-8ec9-f6b80342985d" containerName="payload-pre-processing" containerID="cri-o://ebb4c16b23f14fab56166d51e222dfa982a1de721a6477c3008471e91ddc1cfb" gracePeriod=25
Nov 03 23:06:42.780284 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:42.780183    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:43.831587 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:43.831538    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:43.980294 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:43.980263    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:44.012300 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:44.012261    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-23d281-97b8bbd66-7dgnl" podUID="2452e573-42a1-40b4-b86f-3678286375bc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 23:06:44.013116 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:44.013084    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-23d281-8599fb6555-qtrjc" podUID="98293f18-47d0-49fe-8ec9-f6b80342985d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 23:06:44.092377 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:44.092294    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90310d72-c818-4fdd-b4e7-b51f633caa05" path="/var/lib/kubelet/pods/90310d72-c818-4fdd-b4e7-b51f633caa05/volumes"
Nov 03 23:06:44.147158 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:44.147119    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:44.292896 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:44.292857    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:44.733878 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:44.733845    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:44.887088 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:44.887052    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:45.276686 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:45.276650    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:46.621839 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:46.621803    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:47.754801 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:47.754763    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:47.912829 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:47.912792    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:48.007621 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:48.007524    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-xdjl2"]
Nov 03 23:06:48.007824 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:48.007798    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-xdjl2" podUID="e90b239c-aeaa-4c94-97fb-6c2fd30f9244" containerName="istio-proxy" containerID="cri-o://dd0275646c0e941edc9e0b4ba5c985e4156f809b470dd952bd81054bc86819c3" gracePeriod=30
Nov 03 23:06:48.095982 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:48.095950    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:48.257373 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:48.257334    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:48.910200 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:48.910160    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:49.056573 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:49.056539    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:49.223441 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:49.223405    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:49.437042 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:49.437011    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:49.668647 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:49.668618    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:49.925695 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:49.925595    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:50.862421 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:50.862385    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:51.371159 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:51.371125    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-2d619f-6979556cf5-4sd4b"]
Nov 03 23:06:51.371677 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:51.371404    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-2d619f-6979556cf5-4sd4b" podUID="ab4a1779-440d-442f-bc77-751d89b544c7" containerName="maas-api" containerID="cri-o://fe7b04b4e6db81ae3bb411ad59697f6bfda6e91e474ef92b1dc670f71c80521d" gracePeriod=30
Nov 03 23:06:51.505190 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:51.505159    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:51.641462 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:51.641436    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-2d619f-6979556cf5-4sd4b"
Nov 03 23:06:51.731924 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:51.731890    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab4a1779-440d-442f-bc77-751d89b544c7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab4a1779-440d-442f-bc77-751d89b544c7-openshift-service-ca\") pod \"ab4a1779-440d-442f-bc77-751d89b544c7\" (UID: \"ab4a1779-440d-442f-bc77-751d89b544c7\") "
Nov 03 23:06:51.732109 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:51.731987    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab4a1779-440d-442f-bc77-751d89b544c7-kube-api-access-lh2rz\" (UniqueName: \"kubernetes.io/projected/ab4a1779-440d-442f-bc77-751d89b544c7-kube-api-access-lh2rz\") pod \"ab4a1779-440d-442f-bc77-751d89b544c7\" (UID: \"ab4a1779-440d-442f-bc77-751d89b544c7\") "
Nov 03 23:06:51.732109 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:51.732037    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab4a1779-440d-442f-bc77-751d89b544c7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab4a1779-440d-442f-bc77-751d89b544c7-metrics-certs\") pod \"ab4a1779-440d-442f-bc77-751d89b544c7\" (UID: \"ab4a1779-440d-442f-bc77-751d89b544c7\") "
Nov 03 23:06:51.732199 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:51.732106    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab4a1779-440d-442f-bc77-751d89b544c7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab4a1779-440d-442f-bc77-751d89b544c7-maas-api-tls\") pod \"ab4a1779-440d-442f-bc77-751d89b544c7\" (UID: \"ab4a1779-440d-442f-bc77-751d89b544c7\") "
Nov 03 23:06:51.732346 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:51.732316    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab4a1779-440d-442f-bc77-751d89b544c7-openshift-service-ca" pod "ab4a1779-440d-442f-bc77-751d89b544c7" (UID: "ab4a1779-440d-442f-bc77-751d89b544c7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:06:51.732540 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:51.732518    4431 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab4a1779-440d-442f-bc77-751d89b544c7-openshift-service-ca\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:06:51.734159 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:51.734136    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab4a1779-440d-442f-bc77-751d89b544c7-kube-api-access-lh2rz" pod "ab4a1779-440d-442f-bc77-751d89b544c7" (UID: "ab4a1779-440d-442f-bc77-751d89b544c7"). InnerVolumeSpecName "kube-api-access-lh2rz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:06:51.734288 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:51.734137    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab4a1779-440d-442f-bc77-751d89b544c7-maas-api-tls" pod "ab4a1779-440d-442f-bc77-751d89b544c7" (UID: "ab4a1779-440d-442f-bc77-751d89b544c7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:06:51.734424 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:51.734408    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab4a1779-440d-442f-bc77-751d89b544c7-metrics-certs" pod "ab4a1779-440d-442f-bc77-751d89b544c7" (UID: "ab4a1779-440d-442f-bc77-751d89b544c7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:06:51.751696 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:51.751666    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:51.833521 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:51.833482    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lh2rz\" (UniqueName: \"kubernetes.io/projected/ab4a1779-440d-442f-bc77-751d89b544c7-kube-api-access-lh2rz\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:06:51.833521 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:51.833519    4431 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab4a1779-440d-442f-bc77-751d89b544c7-metrics-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:06:51.833796 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:51.833534    4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab4a1779-440d-442f-bc77-751d89b544c7-maas-api-tls\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:06:51.953355 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:51.953317    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:52.098106 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:52.098072    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:52.150658 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:52.150628    4431 generic.go:356] "Generic (PLEG): container finished" podID="ab4a1779-440d-442f-bc77-751d89b544c7" containerID="fe7b04b4e6db81ae3bb411ad59697f6bfda6e91e474ef92b1dc670f71c80521d" exitCode=0
Nov 03 23:06:52.150833 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:52.150675    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-2d619f-6979556cf5-4sd4b"
Nov 03 23:06:52.150833 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:52.150676    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-2d619f-6979556cf5-4sd4b" event={"ID":"ab4a1779-440d-442f-bc77-751d89b544c7","Type":"ContainerDied","Data":"fe7b04b4e6db81ae3bb411ad59697f6bfda6e91e474ef92b1dc670f71c80521d"}
Nov 03 23:06:52.150833 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:52.150804    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-2d619f-6979556cf5-4sd4b" event={"ID":"ab4a1779-440d-442f-bc77-751d89b544c7","Type":"ContainerDied","Data":"ac95634ae6cc4471861271e7f7bc1fab91744db1a04b10d103907a3de1a3c0c4"}
Nov 03 23:06:52.150833 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:52.150832    4431 scope.go:122] "RemoveContainer" containerID="fe7b04b4e6db81ae3bb411ad59697f6bfda6e91e474ef92b1dc670f71c80521d"
Nov 03 23:06:52.160111 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:52.160096    4431 scope.go:122] "RemoveContainer" containerID="fe7b04b4e6db81ae3bb411ad59697f6bfda6e91e474ef92b1dc670f71c80521d"
Nov 03 23:06:52.160418 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:06:52.160395    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe7b04b4e6db81ae3bb411ad59697f6bfda6e91e474ef92b1dc670f71c80521d\": container with ID starting with fe7b04b4e6db81ae3bb411ad59697f6bfda6e91e474ef92b1dc670f71c80521d not found: ID does not exist" containerID="fe7b04b4e6db81ae3bb411ad59697f6bfda6e91e474ef92b1dc670f71c80521d"
Nov 03 23:06:52.160508 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:52.160429    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe7b04b4e6db81ae3bb411ad59697f6bfda6e91e474ef92b1dc670f71c80521d"} err="failed to get container status \"fe7b04b4e6db81ae3bb411ad59697f6bfda6e91e474ef92b1dc670f71c80521d\": rpc error: code = NotFound desc = could not find container \"fe7b04b4e6db81ae3bb411ad59697f6bfda6e91e474ef92b1dc670f71c80521d\": container with ID starting with fe7b04b4e6db81ae3bb411ad59697f6bfda6e91e474ef92b1dc670f71c80521d not found: ID does not exist"
Nov 03 23:06:52.200622 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:52.200591    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-2d619f-6979556cf5-4sd4b"]
Nov 03 23:06:52.212621 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:52.212556    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-2d619f-6979556cf5-4sd4b"]
Nov 03 23:06:52.249936 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:52.249901    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:52.642407 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:52.642302    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:53.180547 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:53.180505    4431 generic.go:356] "Generic (PLEG): container finished" podID="e90b239c-aeaa-4c94-97fb-6c2fd30f9244" containerID="dd0275646c0e941edc9e0b4ba5c985e4156f809b470dd952bd81054bc86819c3" exitCode=0
Nov 03 23:06:53.180724 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:53.180697    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-xdjl2" event={"ID":"e90b239c-aeaa-4c94-97fb-6c2fd30f9244","Type":"ContainerDied","Data":"dd0275646c0e941edc9e0b4ba5c985e4156f809b470dd952bd81054bc86819c3"}
Nov 03 23:06:53.288978 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:53.288914    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-xdjl2"
Nov 03 23:06:53.454370 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:53.454326    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e90b239c-aeaa-4c94-97fb-6c2fd30f9244-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e90b239c-aeaa-4c94-97fb-6c2fd30f9244-workload-socket\") pod \"e90b239c-aeaa-4c94-97fb-6c2fd30f9244\" (UID: \"e90b239c-aeaa-4c94-97fb-6c2fd30f9244\") "
Nov 03 23:06:53.454583 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:53.454381    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e90b239c-aeaa-4c94-97fb-6c2fd30f9244-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e90b239c-aeaa-4c94-97fb-6c2fd30f9244-credential-socket\") pod \"e90b239c-aeaa-4c94-97fb-6c2fd30f9244\" (UID: \"e90b239c-aeaa-4c94-97fb-6c2fd30f9244\") "
Nov 03 23:06:53.454583 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:53.454432    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e90b239c-aeaa-4c94-97fb-6c2fd30f9244-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e90b239c-aeaa-4c94-97fb-6c2fd30f9244-istiod-ca-cert\") pod \"e90b239c-aeaa-4c94-97fb-6c2fd30f9244\" (UID: \"e90b239c-aeaa-4c94-97fb-6c2fd30f9244\") "
Nov 03 23:06:53.454583 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:53.454482    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e90b239c-aeaa-4c94-97fb-6c2fd30f9244-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e90b239c-aeaa-4c94-97fb-6c2fd30f9244-istio-podinfo\") pod \"e90b239c-aeaa-4c94-97fb-6c2fd30f9244\" (UID: \"e90b239c-aeaa-4c94-97fb-6c2fd30f9244\") "
Nov 03 23:06:53.454583 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:53.454525    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e90b239c-aeaa-4c94-97fb-6c2fd30f9244-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e90b239c-aeaa-4c94-97fb-6c2fd30f9244-istio-envoy\") pod \"e90b239c-aeaa-4c94-97fb-6c2fd30f9244\" (UID: \"e90b239c-aeaa-4c94-97fb-6c2fd30f9244\") "
Nov 03 23:06:53.454834 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:53.454590    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e90b239c-aeaa-4c94-97fb-6c2fd30f9244-istio-token\" (UniqueName: \"kubernetes.io/projected/e90b239c-aeaa-4c94-97fb-6c2fd30f9244-istio-token\") pod \"e90b239c-aeaa-4c94-97fb-6c2fd30f9244\" (UID: \"e90b239c-aeaa-4c94-97fb-6c2fd30f9244\") "
Nov 03 23:06:53.454834 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:53.454676    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e90b239c-aeaa-4c94-97fb-6c2fd30f9244-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e90b239c-aeaa-4c94-97fb-6c2fd30f9244-istio-data\") pod \"e90b239c-aeaa-4c94-97fb-6c2fd30f9244\" (UID: \"e90b239c-aeaa-4c94-97fb-6c2fd30f9244\") "
Nov 03 23:06:53.454834 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:53.454715    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e90b239c-aeaa-4c94-97fb-6c2fd30f9244-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e90b239c-aeaa-4c94-97fb-6c2fd30f9244-workload-certs\") pod \"e90b239c-aeaa-4c94-97fb-6c2fd30f9244\" (UID: \"e90b239c-aeaa-4c94-97fb-6c2fd30f9244\") "
Nov 03 23:06:53.454834 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:53.454759    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e90b239c-aeaa-4c94-97fb-6c2fd30f9244-workload-socket" pod "e90b239c-aeaa-4c94-97fb-6c2fd30f9244" (UID: "e90b239c-aeaa-4c94-97fb-6c2fd30f9244"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:06:53.454834 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:53.454795    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e90b239c-aeaa-4c94-97fb-6c2fd30f9244-kube-api-access-j5zr4\" (UniqueName: \"kubernetes.io/projected/e90b239c-aeaa-4c94-97fb-6c2fd30f9244-kube-api-access-j5zr4\") pod \"e90b239c-aeaa-4c94-97fb-6c2fd30f9244\" (UID: \"e90b239c-aeaa-4c94-97fb-6c2fd30f9244\") "
Nov 03 23:06:53.454834 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:53.454802    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e90b239c-aeaa-4c94-97fb-6c2fd30f9244-credential-socket" pod "e90b239c-aeaa-4c94-97fb-6c2fd30f9244" (UID: "e90b239c-aeaa-4c94-97fb-6c2fd30f9244"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:06:53.455203 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:53.455181    4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e90b239c-aeaa-4c94-97fb-6c2fd30f9244-workload-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:06:53.455303 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:53.455209    4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e90b239c-aeaa-4c94-97fb-6c2fd30f9244-credential-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:06:53.455371 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:53.455350    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e90b239c-aeaa-4c94-97fb-6c2fd30f9244-istio-data" pod "e90b239c-aeaa-4c94-97fb-6c2fd30f9244" (UID: "e90b239c-aeaa-4c94-97fb-6c2fd30f9244"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:06:53.455438 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:53.455377    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e90b239c-aeaa-4c94-97fb-6c2fd30f9244-istiod-ca-cert" pod "e90b239c-aeaa-4c94-97fb-6c2fd30f9244" (UID: "e90b239c-aeaa-4c94-97fb-6c2fd30f9244"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:06:53.455996 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:53.455971    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e90b239c-aeaa-4c94-97fb-6c2fd30f9244-workload-certs" pod "e90b239c-aeaa-4c94-97fb-6c2fd30f9244" (UID: "e90b239c-aeaa-4c94-97fb-6c2fd30f9244"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:06:53.462398 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:53.462366    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e90b239c-aeaa-4c94-97fb-6c2fd30f9244-istio-envoy" pod "e90b239c-aeaa-4c94-97fb-6c2fd30f9244" (UID: "e90b239c-aeaa-4c94-97fb-6c2fd30f9244"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:06:53.462546 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:53.462477    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e90b239c-aeaa-4c94-97fb-6c2fd30f9244-kube-api-access-j5zr4" pod "e90b239c-aeaa-4c94-97fb-6c2fd30f9244" (UID: "e90b239c-aeaa-4c94-97fb-6c2fd30f9244"). InnerVolumeSpecName "kube-api-access-j5zr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:06:53.462688 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:53.462664    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e90b239c-aeaa-4c94-97fb-6c2fd30f9244-istio-token" pod "e90b239c-aeaa-4c94-97fb-6c2fd30f9244" (UID: "e90b239c-aeaa-4c94-97fb-6c2fd30f9244"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:06:53.462783 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:53.462766    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e90b239c-aeaa-4c94-97fb-6c2fd30f9244-istio-podinfo" pod "e90b239c-aeaa-4c94-97fb-6c2fd30f9244" (UID: "e90b239c-aeaa-4c94-97fb-6c2fd30f9244"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 23:06:53.556181 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:53.556080    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5zr4\" (UniqueName: \"kubernetes.io/projected/e90b239c-aeaa-4c94-97fb-6c2fd30f9244-kube-api-access-j5zr4\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:06:53.556181 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:53.556125    4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e90b239c-aeaa-4c94-97fb-6c2fd30f9244-istiod-ca-cert\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:06:53.556181 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:53.556140    4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e90b239c-aeaa-4c94-97fb-6c2fd30f9244-istio-podinfo\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:06:53.556181 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:53.556152    4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e90b239c-aeaa-4c94-97fb-6c2fd30f9244-istio-envoy\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:06:53.556181 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:53.556164    4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e90b239c-aeaa-4c94-97fb-6c2fd30f9244-istio-token\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:06:53.556181 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:53.556177    4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e90b239c-aeaa-4c94-97fb-6c2fd30f9244-istio-data\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:06:53.556181 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:53.556190    4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e90b239c-aeaa-4c94-97fb-6c2fd30f9244-workload-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:06:53.571189 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:53.571156    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:53.723517 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:53.723477    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:53.819564 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:53.819474    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm"]
Nov 03 23:06:53.819761 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:53.819679    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm" podUID="44002d96-03c6-4063-b992-a531d4fe8016" containerName="istio-proxy" containerID="cri-o://19c94ca123cc93fdc602f466ad17a5424546cee8395eec548c6d2c4a216cd062" gracePeriod=30
Nov 03 23:06:53.883785 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:53.883753    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:54.011299 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:54.011256    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-23d281-97b8bbd66-7dgnl" podUID="2452e573-42a1-40b4-b86f-3678286375bc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 23:06:54.011299 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:54.011269    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-23d281-8599fb6555-qtrjc" podUID="98293f18-47d0-49fe-8ec9-f6b80342985d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 23:06:54.033496 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:54.033467    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:54.087142 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:54.087058    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab4a1779-440d-442f-bc77-751d89b544c7" path="/var/lib/kubelet/pods/ab4a1779-440d-442f-bc77-751d89b544c7/volumes"
Nov 03 23:06:54.187431 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:54.187395    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-xdjl2"
Nov 03 23:06:54.187431 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:54.187413    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-xdjl2" event={"ID":"e90b239c-aeaa-4c94-97fb-6c2fd30f9244","Type":"ContainerDied","Data":"052bd0431e7e1ed2878a230804870b7c31d6a9ca8a7fc7b4bf3c03fc1b159f88"}
Nov 03 23:06:54.187692 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:54.187485    4431 scope.go:122] "RemoveContainer" containerID="dd0275646c0e941edc9e0b4ba5c985e4156f809b470dd952bd81054bc86819c3"
Nov 03 23:06:54.242049 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:54.242009    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-xdjl2"]
Nov 03 23:06:54.253956 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:54.253928    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-xdjl2"]
Nov 03 23:06:54.343884 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:54.343805    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:54.586328 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:54.586279    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-2d619f-6979556cf5-4sd4b" podUID="ab4a1779-440d-442f-bc77-751d89b544c7" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.37:8443/health\": context deadline exceeded"
Nov 03 23:06:55.904173 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:55.904142    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:56.043624 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:56.043587    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:56.086859 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:56.086827    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e90b239c-aeaa-4c94-97fb-6c2fd30f9244" path="/var/lib/kubelet/pods/e90b239c-aeaa-4c94-97fb-6c2fd30f9244/volumes"
Nov 03 23:06:57.603746 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:57.603717    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:58.288934 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:58.288900    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:58.561006 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:58.560919    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:58.948510 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:58.948482    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-d8a3d1-85f887b8c4-gfs5q"]
Nov 03 23:06:59.081164 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:59.081133    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm"
Nov 03 23:06:59.132269 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:59.132170    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:59.212865 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:59.212832    4431 generic.go:356] "Generic (PLEG): container finished" podID="44002d96-03c6-4063-b992-a531d4fe8016" containerID="19c94ca123cc93fdc602f466ad17a5424546cee8395eec548c6d2c4a216cd062" exitCode=0
Nov 03 23:06:59.213045 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:59.212879    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm"
Nov 03 23:06:59.213045 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:59.212912    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm" event={"ID":"44002d96-03c6-4063-b992-a531d4fe8016","Type":"ContainerDied","Data":"19c94ca123cc93fdc602f466ad17a5424546cee8395eec548c6d2c4a216cd062"}
Nov 03 23:06:59.213045 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:59.212970    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm" event={"ID":"44002d96-03c6-4063-b992-a531d4fe8016","Type":"ContainerDied","Data":"c8baf491511f5b7a97410689fea32f466877f713feb51532ef78f7ecedbdb7b4"}
Nov 03 23:06:59.213045 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:59.212985    4431 scope.go:122] "RemoveContainer" containerID="19c94ca123cc93fdc602f466ad17a5424546cee8395eec548c6d2c4a216cd062"
Nov 03 23:06:59.216423 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:59.216402    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44002d96-03c6-4063-b992-a531d4fe8016-kube-api-access-492tn\" (UniqueName: \"kubernetes.io/projected/44002d96-03c6-4063-b992-a531d4fe8016-kube-api-access-492tn\") pod \"44002d96-03c6-4063-b992-a531d4fe8016\" (UID: \"44002d96-03c6-4063-b992-a531d4fe8016\") "
Nov 03 23:06:59.216522 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:59.216435    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44002d96-03c6-4063-b992-a531d4fe8016-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44002d96-03c6-4063-b992-a531d4fe8016-credential-socket\") pod \"44002d96-03c6-4063-b992-a531d4fe8016\" (UID: \"44002d96-03c6-4063-b992-a531d4fe8016\") "
Nov 03 23:06:59.216522 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:59.216478    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44002d96-03c6-4063-b992-a531d4fe8016-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44002d96-03c6-4063-b992-a531d4fe8016-istio-data\") pod \"44002d96-03c6-4063-b992-a531d4fe8016\" (UID: \"44002d96-03c6-4063-b992-a531d4fe8016\") "
Nov 03 23:06:59.216522 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:59.216500    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44002d96-03c6-4063-b992-a531d4fe8016-istio-token\" (UniqueName: \"kubernetes.io/projected/44002d96-03c6-4063-b992-a531d4fe8016-istio-token\") pod \"44002d96-03c6-4063-b992-a531d4fe8016\" (UID: \"44002d96-03c6-4063-b992-a531d4fe8016\") "
Nov 03 23:06:59.216644 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:59.216576    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/44002d96-03c6-4063-b992-a531d4fe8016-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44002d96-03c6-4063-b992-a531d4fe8016-istio-podinfo\") pod \"44002d96-03c6-4063-b992-a531d4fe8016\" (UID: \"44002d96-03c6-4063-b992-a531d4fe8016\") "
Nov 03 23:06:59.216644 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:59.216603    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44002d96-03c6-4063-b992-a531d4fe8016-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44002d96-03c6-4063-b992-a531d4fe8016-istiod-ca-cert\") pod \"44002d96-03c6-4063-b992-a531d4fe8016\" (UID: \"44002d96-03c6-4063-b992-a531d4fe8016\") "
Nov 03 23:06:59.216644 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:59.216632    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44002d96-03c6-4063-b992-a531d4fe8016-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44002d96-03c6-4063-b992-a531d4fe8016-workload-socket\") pod \"44002d96-03c6-4063-b992-a531d4fe8016\" (UID: \"44002d96-03c6-4063-b992-a531d4fe8016\") "
Nov 03 23:06:59.216799 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:59.216665    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44002d96-03c6-4063-b992-a531d4fe8016-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44002d96-03c6-4063-b992-a531d4fe8016-istio-envoy\") pod \"44002d96-03c6-4063-b992-a531d4fe8016\" (UID: \"44002d96-03c6-4063-b992-a531d4fe8016\") "
Nov 03 23:06:59.216799 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:59.216704    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44002d96-03c6-4063-b992-a531d4fe8016-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44002d96-03c6-4063-b992-a531d4fe8016-workload-certs\") pod \"44002d96-03c6-4063-b992-a531d4fe8016\" (UID: \"44002d96-03c6-4063-b992-a531d4fe8016\") "
Nov 03 23:06:59.216918 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:59.216809    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44002d96-03c6-4063-b992-a531d4fe8016-credential-socket" pod "44002d96-03c6-4063-b992-a531d4fe8016" (UID: "44002d96-03c6-4063-b992-a531d4fe8016"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:06:59.216918 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:59.216826    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44002d96-03c6-4063-b992-a531d4fe8016-istio-data" pod "44002d96-03c6-4063-b992-a531d4fe8016" (UID: "44002d96-03c6-4063-b992-a531d4fe8016"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:06:59.217092 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:59.217042    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44002d96-03c6-4063-b992-a531d4fe8016-workload-socket" pod "44002d96-03c6-4063-b992-a531d4fe8016" (UID: "44002d96-03c6-4063-b992-a531d4fe8016"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:06:59.217092 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:59.217066    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44002d96-03c6-4063-b992-a531d4fe8016-istiod-ca-cert" pod "44002d96-03c6-4063-b992-a531d4fe8016" (UID: "44002d96-03c6-4063-b992-a531d4fe8016"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:06:59.217420 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:59.217128    4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44002d96-03c6-4063-b992-a531d4fe8016-credential-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:06:59.217420 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:59.217150    4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44002d96-03c6-4063-b992-a531d4fe8016-istio-data\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:06:59.217420 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:59.217311    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44002d96-03c6-4063-b992-a531d4fe8016-workload-certs" pod "44002d96-03c6-4063-b992-a531d4fe8016" (UID: "44002d96-03c6-4063-b992-a531d4fe8016"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:06:59.219068 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:59.219045    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44002d96-03c6-4063-b992-a531d4fe8016-istio-envoy" pod "44002d96-03c6-4063-b992-a531d4fe8016" (UID: "44002d96-03c6-4063-b992-a531d4fe8016"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:06:59.219286 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:59.219261    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44002d96-03c6-4063-b992-a531d4fe8016-istio-token" pod "44002d96-03c6-4063-b992-a531d4fe8016" (UID: "44002d96-03c6-4063-b992-a531d4fe8016"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:06:59.219389 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:59.219367    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/44002d96-03c6-4063-b992-a531d4fe8016-istio-podinfo" pod "44002d96-03c6-4063-b992-a531d4fe8016" (UID: "44002d96-03c6-4063-b992-a531d4fe8016"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 23:06:59.219824 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:59.219651    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44002d96-03c6-4063-b992-a531d4fe8016-kube-api-access-492tn" pod "44002d96-03c6-4063-b992-a531d4fe8016" (UID: "44002d96-03c6-4063-b992-a531d4fe8016"). InnerVolumeSpecName "kube-api-access-492tn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:06:59.223491 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:59.223472    4431 scope.go:122] "RemoveContainer" containerID="19c94ca123cc93fdc602f466ad17a5424546cee8395eec548c6d2c4a216cd062"
Nov 03 23:06:59.223805 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:06:59.223778    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19c94ca123cc93fdc602f466ad17a5424546cee8395eec548c6d2c4a216cd062\": container with ID starting with 19c94ca123cc93fdc602f466ad17a5424546cee8395eec548c6d2c4a216cd062 not found: ID does not exist" containerID="19c94ca123cc93fdc602f466ad17a5424546cee8395eec548c6d2c4a216cd062"
Nov 03 23:06:59.223870 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:59.223816    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19c94ca123cc93fdc602f466ad17a5424546cee8395eec548c6d2c4a216cd062"} err="failed to get container status \"19c94ca123cc93fdc602f466ad17a5424546cee8395eec548c6d2c4a216cd062\": rpc error: code = NotFound desc = could not find container \"19c94ca123cc93fdc602f466ad17a5424546cee8395eec548c6d2c4a216cd062\": container with ID starting with 19c94ca123cc93fdc602f466ad17a5424546cee8395eec548c6d2c4a216cd062 not found: ID does not exist"
Nov 03 23:06:59.318496 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:59.318459    4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44002d96-03c6-4063-b992-a531d4fe8016-istio-token\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:06:59.318496 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:59.318491    4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44002d96-03c6-4063-b992-a531d4fe8016-istio-podinfo\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:06:59.318496 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:59.318501    4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44002d96-03c6-4063-b992-a531d4fe8016-istiod-ca-cert\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:06:59.318767 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:59.318509    4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44002d96-03c6-4063-b992-a531d4fe8016-workload-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:06:59.318767 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:59.318517    4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44002d96-03c6-4063-b992-a531d4fe8016-istio-envoy\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:06:59.318767 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:59.318525    4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44002d96-03c6-4063-b992-a531d4fe8016-workload-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:06:59.318767 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:59.318534    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-492tn\" (UniqueName: \"kubernetes.io/projected/44002d96-03c6-4063-b992-a531d4fe8016-kube-api-access-492tn\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:06:59.339352 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:59.339321    4431 scope.go:122] "RemoveContainer" containerID="5743d436771d83fb1a53146f7f3ccd9f91edd6d7e37618f8241ba0ab25411c46"
Nov 03 23:06:59.340446 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:59.340354    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:59.354061 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:59.354037    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-88a4d9/premium-w0-88a4d9-kserve-5559d75b7b-vqllq"]
Nov 03 23:06:59.363924 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:59.363900    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-88a4d9/simulator-w0-88a4d9-kserve-785f84888f-sjbzp"]
Nov 03 23:06:59.485784 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:59.485748    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:59.557057 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:59.557028    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm"]
Nov 03 23:06:59.568338 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:59.568309    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-4rhzm"]
Nov 03 23:06:59.625457 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:59.625428    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:06:59.869243 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:59.868302    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr"]
Nov 03 23:06:59.869243 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:59.868545    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr" podUID="ffc24874-f9cc-4433-a4f4-2a72788b1067" containerName="istio-proxy" containerID="cri-o://655e474eb3e10ed5bd9595ca30aab9315af6f3fd42c43190a492d94f93d13df8" gracePeriod=30
Nov 03 23:06:59.925772 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:06:59.925730    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:07:00.086475 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:00.086442    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44002d96-03c6-4063-b992-a531d4fe8016" path="/var/lib/kubelet/pods/44002d96-03c6-4063-b992-a531d4fe8016/volumes"
Nov 03 23:07:00.264263 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:00.264211    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:07:00.403323 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:00.403267    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:07:00.912416 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:00.912380    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:07:01.258709 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:07:01.258627    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 23:07:02.518961 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:02.518926    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:07:02.792142 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:02.792118    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-23d281-97b8bbd66-7dgnl"
Nov 03 23:07:02.819013 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:02.818989    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-23d281-8599fb6555-qtrjc"
Nov 03 23:07:02.855918 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:02.855889    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98293f18-47d0-49fe-8ec9-f6b80342985d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98293f18-47d0-49fe-8ec9-f6b80342985d-plugins-config-volume\") pod \"98293f18-47d0-49fe-8ec9-f6b80342985d\" (UID: \"98293f18-47d0-49fe-8ec9-f6b80342985d\") "
Nov 03 23:07:02.856119 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:02.855967    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2452e573-42a1-40b4-b86f-3678286375bc-kube-api-access-wt4v2\" (UniqueName: \"kubernetes.io/projected/2452e573-42a1-40b4-b86f-3678286375bc-kube-api-access-wt4v2\") pod \"2452e573-42a1-40b4-b86f-3678286375bc\" (UID: \"2452e573-42a1-40b4-b86f-3678286375bc\") "
Nov 03 23:07:02.856119 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:02.856112    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2452e573-42a1-40b4-b86f-3678286375bc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2452e573-42a1-40b4-b86f-3678286375bc-plugins-config-volume\") pod \"2452e573-42a1-40b4-b86f-3678286375bc\" (UID: \"2452e573-42a1-40b4-b86f-3678286375bc\") "
Nov 03 23:07:02.856277 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:02.856145    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98293f18-47d0-49fe-8ec9-f6b80342985d-kube-api-access-77hll\" (UniqueName: \"kubernetes.io/projected/98293f18-47d0-49fe-8ec9-f6b80342985d-kube-api-access-77hll\") pod \"98293f18-47d0-49fe-8ec9-f6b80342985d\" (UID: \"98293f18-47d0-49fe-8ec9-f6b80342985d\") "
Nov 03 23:07:02.856429 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:02.856398    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98293f18-47d0-49fe-8ec9-f6b80342985d-plugins-config-volume" pod "98293f18-47d0-49fe-8ec9-f6b80342985d" (UID: "98293f18-47d0-49fe-8ec9-f6b80342985d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:07:02.856701 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:02.856672    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2452e573-42a1-40b4-b86f-3678286375bc-plugins-config-volume" pod "2452e573-42a1-40b4-b86f-3678286375bc" (UID: "2452e573-42a1-40b4-b86f-3678286375bc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:07:02.856810 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:02.856704    4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98293f18-47d0-49fe-8ec9-f6b80342985d-plugins-config-volume\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:07:02.858323 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:02.858300    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2452e573-42a1-40b4-b86f-3678286375bc-kube-api-access-wt4v2" pod "2452e573-42a1-40b4-b86f-3678286375bc" (UID: "2452e573-42a1-40b4-b86f-3678286375bc"). InnerVolumeSpecName "kube-api-access-wt4v2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:07:02.858432 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:02.858372    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98293f18-47d0-49fe-8ec9-f6b80342985d-kube-api-access-77hll" pod "98293f18-47d0-49fe-8ec9-f6b80342985d" (UID: "98293f18-47d0-49fe-8ec9-f6b80342985d"). InnerVolumeSpecName "kube-api-access-77hll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:07:02.957896 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:02.957856    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wt4v2\" (UniqueName: \"kubernetes.io/projected/2452e573-42a1-40b4-b86f-3678286375bc-kube-api-access-wt4v2\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:07:02.957896 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:02.957889    4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2452e573-42a1-40b4-b86f-3678286375bc-plugins-config-volume\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:07:02.957896 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:02.957901    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77hll\" (UniqueName: \"kubernetes.io/projected/98293f18-47d0-49fe-8ec9-f6b80342985d-kube-api-access-77hll\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:07:03.029421 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:03.029341    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:07:03.165385 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:03.165351    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:07:03.234876 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:03.234839    4431 generic.go:356] "Generic (PLEG): container finished" podID="2452e573-42a1-40b4-b86f-3678286375bc" containerID="936af0258adcab8270e0bc34efc0e1efc8bbc52867a87acbb0672153ba8166b5" exitCode=0
Nov 03 23:07:03.235059 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:03.234887    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-23d281-97b8bbd66-7dgnl"
Nov 03 23:07:03.235059 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:03.234920    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-23d281-97b8bbd66-7dgnl" event={"ID":"2452e573-42a1-40b4-b86f-3678286375bc","Type":"ContainerDied","Data":"936af0258adcab8270e0bc34efc0e1efc8bbc52867a87acbb0672153ba8166b5"}
Nov 03 23:07:03.235059 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:03.234972    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-23d281-97b8bbd66-7dgnl" event={"ID":"2452e573-42a1-40b4-b86f-3678286375bc","Type":"ContainerDied","Data":"8cec14553c18152ee52c4f4aa1f6b7406d895fe880087495a0d0d6f4caa38ccc"}
Nov 03 23:07:03.235059 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:03.235002    4431 scope.go:122] "RemoveContainer" containerID="936af0258adcab8270e0bc34efc0e1efc8bbc52867a87acbb0672153ba8166b5"
Nov 03 23:07:03.236404 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:03.236380    4431 generic.go:356] "Generic (PLEG): container finished" podID="98293f18-47d0-49fe-8ec9-f6b80342985d" containerID="ebb4c16b23f14fab56166d51e222dfa982a1de721a6477c3008471e91ddc1cfb" exitCode=0
Nov 03 23:07:03.236530 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:03.236444    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-23d281-8599fb6555-qtrjc"
Nov 03 23:07:03.236530 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:03.236464    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-23d281-8599fb6555-qtrjc" event={"ID":"98293f18-47d0-49fe-8ec9-f6b80342985d","Type":"ContainerDied","Data":"ebb4c16b23f14fab56166d51e222dfa982a1de721a6477c3008471e91ddc1cfb"}
Nov 03 23:07:03.236530 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:03.236514    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-23d281-8599fb6555-qtrjc" event={"ID":"98293f18-47d0-49fe-8ec9-f6b80342985d","Type":"ContainerDied","Data":"b6df291bf775b70b0549e600889c510de72352b11a99eb1c38bf86ed52c344f7"}
Nov 03 23:07:03.246774 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:03.246758    4431 scope.go:122] "RemoveContainer" containerID="936af0258adcab8270e0bc34efc0e1efc8bbc52867a87acbb0672153ba8166b5"
Nov 03 23:07:03.247067 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:07:03.247042    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"936af0258adcab8270e0bc34efc0e1efc8bbc52867a87acbb0672153ba8166b5\": container with ID starting with 936af0258adcab8270e0bc34efc0e1efc8bbc52867a87acbb0672153ba8166b5 not found: ID does not exist" containerID="936af0258adcab8270e0bc34efc0e1efc8bbc52867a87acbb0672153ba8166b5"
Nov 03 23:07:03.247140 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:03.247079    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"936af0258adcab8270e0bc34efc0e1efc8bbc52867a87acbb0672153ba8166b5"} err="failed to get container status \"936af0258adcab8270e0bc34efc0e1efc8bbc52867a87acbb0672153ba8166b5\": rpc error: code = NotFound desc = could not find container \"936af0258adcab8270e0bc34efc0e1efc8bbc52867a87acbb0672153ba8166b5\": container with ID starting with 936af0258adcab8270e0bc34efc0e1efc8bbc52867a87acbb0672153ba8166b5 not found: ID does not exist"
Nov 03 23:07:03.247140 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:03.247097    4431 scope.go:122] "RemoveContainer" containerID="ebb4c16b23f14fab56166d51e222dfa982a1de721a6477c3008471e91ddc1cfb"
Nov 03 23:07:03.255744 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:03.255729    4431 scope.go:122] "RemoveContainer" containerID="ebb4c16b23f14fab56166d51e222dfa982a1de721a6477c3008471e91ddc1cfb"
Nov 03 23:07:03.256013 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:07:03.255982    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebb4c16b23f14fab56166d51e222dfa982a1de721a6477c3008471e91ddc1cfb\": container with ID starting with ebb4c16b23f14fab56166d51e222dfa982a1de721a6477c3008471e91ddc1cfb not found: ID does not exist" containerID="ebb4c16b23f14fab56166d51e222dfa982a1de721a6477c3008471e91ddc1cfb"
Nov 03 23:07:03.256065 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:03.256017    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebb4c16b23f14fab56166d51e222dfa982a1de721a6477c3008471e91ddc1cfb"} err="failed to get container status \"ebb4c16b23f14fab56166d51e222dfa982a1de721a6477c3008471e91ddc1cfb\": rpc error: code = NotFound desc = could not find container \"ebb4c16b23f14fab56166d51e222dfa982a1de721a6477c3008471e91ddc1cfb\": container with ID starting with ebb4c16b23f14fab56166d51e222dfa982a1de721a6477c3008471e91ddc1cfb not found: ID does not exist"
Nov 03 23:07:03.285208 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:03.285150    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-23d281-8599fb6555-qtrjc"]
Nov 03 23:07:03.296748 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:03.296719    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-23d281-8599fb6555-qtrjc"]
Nov 03 23:07:03.329793 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:03.329762    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-23d281-97b8bbd66-7dgnl"]
Nov 03 23:07:03.338839 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:03.338813    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-23d281-97b8bbd66-7dgnl"]
Nov 03 23:07:03.949830 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:03.949788    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d8a3d1-85f887b8c4-gfs5q" podUID="94198afd-2296-4658-ba19-6624a26f2bf9" containerName="payload-processing" containerID="cri-o://d3109ce13320cddcfdadb8985aa6bb60d472eb6cf56ed093b5485bb2799c820c" gracePeriod=25
Nov 03 23:07:03.986254 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:03.986199    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:07:04.087652 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:04.087619    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2452e573-42a1-40b4-b86f-3678286375bc" path="/var/lib/kubelet/pods/2452e573-42a1-40b4-b86f-3678286375bc/volumes"
Nov 03 23:07:04.088025 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:04.088010    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98293f18-47d0-49fe-8ec9-f6b80342985d" path="/var/lib/kubelet/pods/98293f18-47d0-49fe-8ec9-f6b80342985d/volumes"
Nov 03 23:07:04.509774 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:04.509734    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:07:04.666089 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:04.666051    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:07:05.141196 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:05.140897    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr"
Nov 03 23:07:05.141778 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:05.141752    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:07:05.181498 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:05.181465    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffc24874-f9cc-4433-a4f4-2a72788b1067-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffc24874-f9cc-4433-a4f4-2a72788b1067-workload-certs\") pod \"ffc24874-f9cc-4433-a4f4-2a72788b1067\" (UID: \"ffc24874-f9cc-4433-a4f4-2a72788b1067\") "
Nov 03 23:07:05.181695 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:05.181610    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ffc24874-f9cc-4433-a4f4-2a72788b1067-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffc24874-f9cc-4433-a4f4-2a72788b1067-istio-podinfo\") pod \"ffc24874-f9cc-4433-a4f4-2a72788b1067\" (UID: \"ffc24874-f9cc-4433-a4f4-2a72788b1067\") "
Nov 03 23:07:05.181695 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:05.181649    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffc24874-f9cc-4433-a4f4-2a72788b1067-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffc24874-f9cc-4433-a4f4-2a72788b1067-istio-data\") pod \"ffc24874-f9cc-4433-a4f4-2a72788b1067\" (UID: \"ffc24874-f9cc-4433-a4f4-2a72788b1067\") "
Nov 03 23:07:05.181837 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:05.181733    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffc24874-f9cc-4433-a4f4-2a72788b1067-kube-api-access-fdc4r\" (UniqueName: \"kubernetes.io/projected/ffc24874-f9cc-4433-a4f4-2a72788b1067-kube-api-access-fdc4r\") pod \"ffc24874-f9cc-4433-a4f4-2a72788b1067\" (UID: \"ffc24874-f9cc-4433-a4f4-2a72788b1067\") "
Nov 03 23:07:05.181837 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:05.181773    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffc24874-f9cc-4433-a4f4-2a72788b1067-workload-certs" pod "ffc24874-f9cc-4433-a4f4-2a72788b1067" (UID: "ffc24874-f9cc-4433-a4f4-2a72788b1067"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:07:05.181837 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:05.181791    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffc24874-f9cc-4433-a4f4-2a72788b1067-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffc24874-f9cc-4433-a4f4-2a72788b1067-workload-socket\") pod \"ffc24874-f9cc-4433-a4f4-2a72788b1067\" (UID: \"ffc24874-f9cc-4433-a4f4-2a72788b1067\") "
Nov 03 23:07:05.181837 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:05.181825    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffc24874-f9cc-4433-a4f4-2a72788b1067-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffc24874-f9cc-4433-a4f4-2a72788b1067-credential-socket\") pod \"ffc24874-f9cc-4433-a4f4-2a72788b1067\" (UID: \"ffc24874-f9cc-4433-a4f4-2a72788b1067\") "
Nov 03 23:07:05.182066 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:05.181855    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffc24874-f9cc-4433-a4f4-2a72788b1067-istio-token\" (UniqueName: \"kubernetes.io/projected/ffc24874-f9cc-4433-a4f4-2a72788b1067-istio-token\") pod \"ffc24874-f9cc-4433-a4f4-2a72788b1067\" (UID: \"ffc24874-f9cc-4433-a4f4-2a72788b1067\") "
Nov 03 23:07:05.182066 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:05.181891    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffc24874-f9cc-4433-a4f4-2a72788b1067-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffc24874-f9cc-4433-a4f4-2a72788b1067-istio-envoy\") pod \"ffc24874-f9cc-4433-a4f4-2a72788b1067\" (UID: \"ffc24874-f9cc-4433-a4f4-2a72788b1067\") "
Nov 03 23:07:05.182066 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:05.181932    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffc24874-f9cc-4433-a4f4-2a72788b1067-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffc24874-f9cc-4433-a4f4-2a72788b1067-istiod-ca-cert\") pod \"ffc24874-f9cc-4433-a4f4-2a72788b1067\" (UID: \"ffc24874-f9cc-4433-a4f4-2a72788b1067\") "
Nov 03 23:07:05.182066 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:05.181974    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffc24874-f9cc-4433-a4f4-2a72788b1067-istio-data" pod "ffc24874-f9cc-4433-a4f4-2a72788b1067" (UID: "ffc24874-f9cc-4433-a4f4-2a72788b1067"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:07:05.182375 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:05.182352    4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffc24874-f9cc-4433-a4f4-2a72788b1067-istio-data\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:07:05.182453 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:05.182381    4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffc24874-f9cc-4433-a4f4-2a72788b1067-workload-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:07:05.182546 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:05.182524    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffc24874-f9cc-4433-a4f4-2a72788b1067-credential-socket" pod "ffc24874-f9cc-4433-a4f4-2a72788b1067" (UID: "ffc24874-f9cc-4433-a4f4-2a72788b1067"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:07:05.182638 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:05.182615    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffc24874-f9cc-4433-a4f4-2a72788b1067-workload-socket" pod "ffc24874-f9cc-4433-a4f4-2a72788b1067" (UID: "ffc24874-f9cc-4433-a4f4-2a72788b1067"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:07:05.182739 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:05.182714    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffc24874-f9cc-4433-a4f4-2a72788b1067-istiod-ca-cert" pod "ffc24874-f9cc-4433-a4f4-2a72788b1067" (UID: "ffc24874-f9cc-4433-a4f4-2a72788b1067"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:07:05.185512 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:05.185392    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffc24874-f9cc-4433-a4f4-2a72788b1067-kube-api-access-fdc4r" pod "ffc24874-f9cc-4433-a4f4-2a72788b1067" (UID: "ffc24874-f9cc-4433-a4f4-2a72788b1067"). InnerVolumeSpecName "kube-api-access-fdc4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:07:05.185512 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:05.185481    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ffc24874-f9cc-4433-a4f4-2a72788b1067-istio-podinfo" pod "ffc24874-f9cc-4433-a4f4-2a72788b1067" (UID: "ffc24874-f9cc-4433-a4f4-2a72788b1067"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 23:07:05.186983 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:05.186955    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffc24874-f9cc-4433-a4f4-2a72788b1067-istio-envoy" pod "ffc24874-f9cc-4433-a4f4-2a72788b1067" (UID: "ffc24874-f9cc-4433-a4f4-2a72788b1067"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:07:05.187102 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:05.186984    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffc24874-f9cc-4433-a4f4-2a72788b1067-istio-token" pod "ffc24874-f9cc-4433-a4f4-2a72788b1067" (UID: "ffc24874-f9cc-4433-a4f4-2a72788b1067"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:07:05.248988 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:05.248955    4431 generic.go:356] "Generic (PLEG): container finished" podID="ffc24874-f9cc-4433-a4f4-2a72788b1067" containerID="655e474eb3e10ed5bd9595ca30aab9315af6f3fd42c43190a492d94f93d13df8" exitCode=0
Nov 03 23:07:05.249167 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:05.249002    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr"
Nov 03 23:07:05.249167 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:05.249004    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr" event={"ID":"ffc24874-f9cc-4433-a4f4-2a72788b1067","Type":"ContainerDied","Data":"655e474eb3e10ed5bd9595ca30aab9315af6f3fd42c43190a492d94f93d13df8"}
Nov 03 23:07:05.249167 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:05.249059    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr" event={"ID":"ffc24874-f9cc-4433-a4f4-2a72788b1067","Type":"ContainerDied","Data":"1b62939e106025b1f179bc95ca68191991a0f31d69f592ee5c1be28e2be807fc"}
Nov 03 23:07:05.249167 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:05.249082    4431 scope.go:122] "RemoveContainer" containerID="655e474eb3e10ed5bd9595ca30aab9315af6f3fd42c43190a492d94f93d13df8"
Nov 03 23:07:05.259603 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:05.259584    4431 scope.go:122] "RemoveContainer" containerID="655e474eb3e10ed5bd9595ca30aab9315af6f3fd42c43190a492d94f93d13df8"
Nov 03 23:07:05.259870 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:07:05.259850    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"655e474eb3e10ed5bd9595ca30aab9315af6f3fd42c43190a492d94f93d13df8\": container with ID starting with 655e474eb3e10ed5bd9595ca30aab9315af6f3fd42c43190a492d94f93d13df8 not found: ID does not exist" containerID="655e474eb3e10ed5bd9595ca30aab9315af6f3fd42c43190a492d94f93d13df8"
Nov 03 23:07:05.259917 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:05.259879    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"655e474eb3e10ed5bd9595ca30aab9315af6f3fd42c43190a492d94f93d13df8"} err="failed to get container status \"655e474eb3e10ed5bd9595ca30aab9315af6f3fd42c43190a492d94f93d13df8\": rpc error: code = NotFound desc = could not find container \"655e474eb3e10ed5bd9595ca30aab9315af6f3fd42c43190a492d94f93d13df8\": container with ID starting with 655e474eb3e10ed5bd9595ca30aab9315af6f3fd42c43190a492d94f93d13df8 not found: ID does not exist"
Nov 03 23:07:05.283821 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:05.283792    4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffc24874-f9cc-4433-a4f4-2a72788b1067-istio-podinfo\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:07:05.283821 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:05.283817    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdc4r\" (UniqueName: \"kubernetes.io/projected/ffc24874-f9cc-4433-a4f4-2a72788b1067-kube-api-access-fdc4r\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:07:05.284016 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:05.283828    4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffc24874-f9cc-4433-a4f4-2a72788b1067-workload-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:07:05.284016 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:05.283839    4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffc24874-f9cc-4433-a4f4-2a72788b1067-credential-socket\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:07:05.284016 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:05.283848    4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffc24874-f9cc-4433-a4f4-2a72788b1067-istio-token\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:07:05.284016 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:05.283857    4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffc24874-f9cc-4433-a4f4-2a72788b1067-istio-envoy\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:07:05.284016 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:05.283864    4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffc24874-f9cc-4433-a4f4-2a72788b1067-istiod-ca-cert\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:07:05.309492 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:05.309459    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr"]
Nov 03 23:07:05.318465 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:05.318435    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:07:05.320819 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:05.320783    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-2gnhr"]
Nov 03 23:07:06.082802 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:06.082768    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:07:06.093528 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:06.093494    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffc24874-f9cc-4433-a4f4-2a72788b1067" path="/var/lib/kubelet/pods/ffc24874-f9cc-4433-a4f4-2a72788b1067/volumes"
Nov 03 23:07:06.571384 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:06.571351    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:07:08.233327 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:08.233291    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:07:09.920709 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:09.920666    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:07:11.217125 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:11.217088    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d8a3d1-85f887b8c4-gfs5q" podUID="94198afd-2296-4658-ba19-6624a26f2bf9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 23:07:11.512929 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:11.512850    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:07:12.445012 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:12.444974    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:07:12.655803 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:12.655759    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:07:12.805470 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:12.805366    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:07:13.219376 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:13.219341    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:07:13.370507 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:13.370448    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:07:13.535505 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:13.535421    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:07:14.378353 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:14.378316    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-88a4d9/premium-w0-88a4d9-kserve-5559d75b7b-vqllq" podUID="05206737-4ca3-41fc-b35a-37c625cc182c" containerName="main" containerID="cri-o://d71101cf7f8cac4b13219889ee6bdde6099376f3a2df441098177a241ab2e114" gracePeriod=45
Nov 03 23:07:14.385103 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:14.385073    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-88a4d9/simulator-w0-88a4d9-kserve-785f84888f-sjbzp" podUID="18bd460b-a0f1-415c-bcb4-41c9f70d2926" containerName="main" containerID="cri-o://bf12d317e70bd7804623af7d868541932ca2fc2a29807d96e123bc699d20a7c3" gracePeriod=45
Nov 03 23:07:14.635387 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:14.635324    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-88a4d9/premium-w0-88a4d9-kserve-5559d75b7b-vqllq"
Nov 03 23:07:14.672274 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:14.672244    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-88a4d9/simulator-w0-88a4d9-kserve-785f84888f-sjbzp"
Nov 03 23:07:14.672545 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:14.672528    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05206737-4ca3-41fc-b35a-37c625cc182c-tls-certs\" (UniqueName: \"kubernetes.io/secret/05206737-4ca3-41fc-b35a-37c625cc182c-tls-certs\") pod \"05206737-4ca3-41fc-b35a-37c625cc182c\" (UID: \"05206737-4ca3-41fc-b35a-37c625cc182c\") "
Nov 03 23:07:14.672642 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:14.672628    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05206737-4ca3-41fc-b35a-37c625cc182c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05206737-4ca3-41fc-b35a-37c625cc182c-model-cache\") pod \"05206737-4ca3-41fc-b35a-37c625cc182c\" (UID: \"05206737-4ca3-41fc-b35a-37c625cc182c\") "
Nov 03 23:07:14.672701 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:14.672673    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05206737-4ca3-41fc-b35a-37c625cc182c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05206737-4ca3-41fc-b35a-37c625cc182c-tmp-dir\") pod \"05206737-4ca3-41fc-b35a-37c625cc182c\" (UID: \"05206737-4ca3-41fc-b35a-37c625cc182c\") "
Nov 03 23:07:14.672766 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:14.672751    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05206737-4ca3-41fc-b35a-37c625cc182c-home\" (UniqueName: \"kubernetes.io/empty-dir/05206737-4ca3-41fc-b35a-37c625cc182c-home\") pod \"05206737-4ca3-41fc-b35a-37c625cc182c\" (UID: \"05206737-4ca3-41fc-b35a-37c625cc182c\") "
Nov 03 23:07:14.672845 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:14.672832    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05206737-4ca3-41fc-b35a-37c625cc182c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/05206737-4ca3-41fc-b35a-37c625cc182c-dshm\") pod \"05206737-4ca3-41fc-b35a-37c625cc182c\" (UID: \"05206737-4ca3-41fc-b35a-37c625cc182c\") "
Nov 03 23:07:14.672941 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:14.672926    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05206737-4ca3-41fc-b35a-37c625cc182c-kube-api-access-thvpg\" (UniqueName: \"kubernetes.io/projected/05206737-4ca3-41fc-b35a-37c625cc182c-kube-api-access-thvpg\") pod \"05206737-4ca3-41fc-b35a-37c625cc182c\" (UID: \"05206737-4ca3-41fc-b35a-37c625cc182c\") "
Nov 03 23:07:14.673005 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:14.672936    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05206737-4ca3-41fc-b35a-37c625cc182c-model-cache" pod "05206737-4ca3-41fc-b35a-37c625cc182c" (UID: "05206737-4ca3-41fc-b35a-37c625cc182c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:07:14.673113 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:14.673089    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05206737-4ca3-41fc-b35a-37c625cc182c-home" pod "05206737-4ca3-41fc-b35a-37c625cc182c" (UID: "05206737-4ca3-41fc-b35a-37c625cc182c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:07:14.673452 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:14.673393    4431 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05206737-4ca3-41fc-b35a-37c625cc182c-model-cache\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:07:14.673452 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:14.673420    4431 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05206737-4ca3-41fc-b35a-37c625cc182c-home\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:07:14.673741 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:14.673704    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05206737-4ca3-41fc-b35a-37c625cc182c-tmp-dir" pod "05206737-4ca3-41fc-b35a-37c625cc182c" (UID: "05206737-4ca3-41fc-b35a-37c625cc182c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:07:14.675320 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:14.675291    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05206737-4ca3-41fc-b35a-37c625cc182c-dshm" pod "05206737-4ca3-41fc-b35a-37c625cc182c" (UID: "05206737-4ca3-41fc-b35a-37c625cc182c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:07:14.675445 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:14.675423    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05206737-4ca3-41fc-b35a-37c625cc182c-tls-certs" pod "05206737-4ca3-41fc-b35a-37c625cc182c" (UID: "05206737-4ca3-41fc-b35a-37c625cc182c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:07:14.675513 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:14.675457    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05206737-4ca3-41fc-b35a-37c625cc182c-kube-api-access-thvpg" pod "05206737-4ca3-41fc-b35a-37c625cc182c" (UID: "05206737-4ca3-41fc-b35a-37c625cc182c"). InnerVolumeSpecName "kube-api-access-thvpg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:07:14.774158 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:14.774124    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18bd460b-a0f1-415c-bcb4-41c9f70d2926-kube-api-access-jmbrj\" (UniqueName: \"kubernetes.io/projected/18bd460b-a0f1-415c-bcb4-41c9f70d2926-kube-api-access-jmbrj\") pod \"18bd460b-a0f1-415c-bcb4-41c9f70d2926\" (UID: \"18bd460b-a0f1-415c-bcb4-41c9f70d2926\") "
Nov 03 23:07:14.774158 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:14.774165    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18bd460b-a0f1-415c-bcb4-41c9f70d2926-dshm\" (UniqueName: \"kubernetes.io/empty-dir/18bd460b-a0f1-415c-bcb4-41c9f70d2926-dshm\") pod \"18bd460b-a0f1-415c-bcb4-41c9f70d2926\" (UID: \"18bd460b-a0f1-415c-bcb4-41c9f70d2926\") "
Nov 03 23:07:14.774426 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:14.774260    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18bd460b-a0f1-415c-bcb4-41c9f70d2926-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18bd460b-a0f1-415c-bcb4-41c9f70d2926-model-cache\") pod \"18bd460b-a0f1-415c-bcb4-41c9f70d2926\" (UID: \"18bd460b-a0f1-415c-bcb4-41c9f70d2926\") "
Nov 03 23:07:14.774426 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:14.774314    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18bd460b-a0f1-415c-bcb4-41c9f70d2926-home\" (UniqueName: \"kubernetes.io/empty-dir/18bd460b-a0f1-415c-bcb4-41c9f70d2926-home\") pod \"18bd460b-a0f1-415c-bcb4-41c9f70d2926\" (UID: \"18bd460b-a0f1-415c-bcb4-41c9f70d2926\") "
Nov 03 23:07:14.774426 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:14.774340    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18bd460b-a0f1-415c-bcb4-41c9f70d2926-tls-certs\" (UniqueName: \"kubernetes.io/secret/18bd460b-a0f1-415c-bcb4-41c9f70d2926-tls-certs\") pod \"18bd460b-a0f1-415c-bcb4-41c9f70d2926\" (UID: \"18bd460b-a0f1-415c-bcb4-41c9f70d2926\") "
Nov 03 23:07:14.774599 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:14.774432    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18bd460b-a0f1-415c-bcb4-41c9f70d2926-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18bd460b-a0f1-415c-bcb4-41c9f70d2926-tmp-dir\") pod \"18bd460b-a0f1-415c-bcb4-41c9f70d2926\" (UID: \"18bd460b-a0f1-415c-bcb4-41c9f70d2926\") "
Nov 03 23:07:14.774688 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:14.774652    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18bd460b-a0f1-415c-bcb4-41c9f70d2926-home" pod "18bd460b-a0f1-415c-bcb4-41c9f70d2926" (UID: "18bd460b-a0f1-415c-bcb4-41c9f70d2926"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:07:14.774782 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:14.774751    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18bd460b-a0f1-415c-bcb4-41c9f70d2926-tmp-dir" pod "18bd460b-a0f1-415c-bcb4-41c9f70d2926" (UID: "18bd460b-a0f1-415c-bcb4-41c9f70d2926"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:07:14.774925 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:14.774817    4431 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18bd460b-a0f1-415c-bcb4-41c9f70d2926-home\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:07:14.774925 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:14.774833    4431 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05206737-4ca3-41fc-b35a-37c625cc182c-dshm\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:07:14.774925 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:14.774843    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thvpg\" (UniqueName: \"kubernetes.io/projected/05206737-4ca3-41fc-b35a-37c625cc182c-kube-api-access-thvpg\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:07:14.774925 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:14.774859    4431 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05206737-4ca3-41fc-b35a-37c625cc182c-tls-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:07:14.774925 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:14.774874    4431 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05206737-4ca3-41fc-b35a-37c625cc182c-tmp-dir\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:07:14.774925 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:14.774879    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18bd460b-a0f1-415c-bcb4-41c9f70d2926-model-cache" pod "18bd460b-a0f1-415c-bcb4-41c9f70d2926" (UID: "18bd460b-a0f1-415c-bcb4-41c9f70d2926"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:07:14.776331 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:14.776310    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18bd460b-a0f1-415c-bcb4-41c9f70d2926-dshm" pod "18bd460b-a0f1-415c-bcb4-41c9f70d2926" (UID: "18bd460b-a0f1-415c-bcb4-41c9f70d2926"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:07:14.776404 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:14.776386    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18bd460b-a0f1-415c-bcb4-41c9f70d2926-kube-api-access-jmbrj" pod "18bd460b-a0f1-415c-bcb4-41c9f70d2926" (UID: "18bd460b-a0f1-415c-bcb4-41c9f70d2926"). InnerVolumeSpecName "kube-api-access-jmbrj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:07:14.776455 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:14.776428    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18bd460b-a0f1-415c-bcb4-41c9f70d2926-tls-certs" pod "18bd460b-a0f1-415c-bcb4-41c9f70d2926" (UID: "18bd460b-a0f1-415c-bcb4-41c9f70d2926"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:07:14.875383 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:14.875346    4431 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18bd460b-a0f1-415c-bcb4-41c9f70d2926-tls-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:07:14.875383 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:14.875376    4431 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18bd460b-a0f1-415c-bcb4-41c9f70d2926-tmp-dir\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:07:14.875383 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:14.875385    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmbrj\" (UniqueName: \"kubernetes.io/projected/18bd460b-a0f1-415c-bcb4-41c9f70d2926-kube-api-access-jmbrj\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:07:14.875617 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:14.875394    4431 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18bd460b-a0f1-415c-bcb4-41c9f70d2926-dshm\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:07:14.875617 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:14.875403    4431 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18bd460b-a0f1-415c-bcb4-41c9f70d2926-model-cache\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:07:15.000980 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:15.000947    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:07:15.299359 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:15.299268    4431 generic.go:356] "Generic (PLEG): container finished" podID="18bd460b-a0f1-415c-bcb4-41c9f70d2926" containerID="bf12d317e70bd7804623af7d868541932ca2fc2a29807d96e123bc699d20a7c3" exitCode=0
Nov 03 23:07:15.299359 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:15.299319    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-88a4d9/simulator-w0-88a4d9-kserve-785f84888f-sjbzp"
Nov 03 23:07:15.299599 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:15.299368    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-88a4d9/simulator-w0-88a4d9-kserve-785f84888f-sjbzp" event={"ID":"18bd460b-a0f1-415c-bcb4-41c9f70d2926","Type":"ContainerDied","Data":"bf12d317e70bd7804623af7d868541932ca2fc2a29807d96e123bc699d20a7c3"}
Nov 03 23:07:15.299599 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:15.299417    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-88a4d9/simulator-w0-88a4d9-kserve-785f84888f-sjbzp" event={"ID":"18bd460b-a0f1-415c-bcb4-41c9f70d2926","Type":"ContainerDied","Data":"41c518a53ca4870bbf7e9db66c21461c685199dc220ae1b28a26f0df78ffa859"}
Nov 03 23:07:15.299599 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:15.299441    4431 scope.go:122] "RemoveContainer" containerID="bf12d317e70bd7804623af7d868541932ca2fc2a29807d96e123bc699d20a7c3"
Nov 03 23:07:15.300778 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:15.300757    4431 generic.go:356] "Generic (PLEG): container finished" podID="05206737-4ca3-41fc-b35a-37c625cc182c" containerID="d71101cf7f8cac4b13219889ee6bdde6099376f3a2df441098177a241ab2e114" exitCode=0
Nov 03 23:07:15.300875 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:15.300809    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-88a4d9/premium-w0-88a4d9-kserve-5559d75b7b-vqllq"
Nov 03 23:07:15.300875 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:15.300820    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-88a4d9/premium-w0-88a4d9-kserve-5559d75b7b-vqllq" event={"ID":"05206737-4ca3-41fc-b35a-37c625cc182c","Type":"ContainerDied","Data":"d71101cf7f8cac4b13219889ee6bdde6099376f3a2df441098177a241ab2e114"}
Nov 03 23:07:15.300875 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:15.300851    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-88a4d9/premium-w0-88a4d9-kserve-5559d75b7b-vqllq" event={"ID":"05206737-4ca3-41fc-b35a-37c625cc182c","Type":"ContainerDied","Data":"76dc9f00e4974208cb67becda0970aa924d111b0553f24e694a25bbc03b28193"}
Nov 03 23:07:15.308282 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:15.308257    4431 scope.go:122] "RemoveContainer" containerID="bf12d317e70bd7804623af7d868541932ca2fc2a29807d96e123bc699d20a7c3"
Nov 03 23:07:15.308714 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:07:15.308693    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf12d317e70bd7804623af7d868541932ca2fc2a29807d96e123bc699d20a7c3\": container with ID starting with bf12d317e70bd7804623af7d868541932ca2fc2a29807d96e123bc699d20a7c3 not found: ID does not exist" containerID="bf12d317e70bd7804623af7d868541932ca2fc2a29807d96e123bc699d20a7c3"
Nov 03 23:07:15.308816 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:15.308721    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf12d317e70bd7804623af7d868541932ca2fc2a29807d96e123bc699d20a7c3"} err="failed to get container status \"bf12d317e70bd7804623af7d868541932ca2fc2a29807d96e123bc699d20a7c3\": rpc error: code = NotFound desc = could not find container \"bf12d317e70bd7804623af7d868541932ca2fc2a29807d96e123bc699d20a7c3\": container with ID starting with bf12d317e70bd7804623af7d868541932ca2fc2a29807d96e123bc699d20a7c3 not found: ID does not exist"
Nov 03 23:07:15.308816 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:15.308739    4431 scope.go:122] "RemoveContainer" containerID="d71101cf7f8cac4b13219889ee6bdde6099376f3a2df441098177a241ab2e114"
Nov 03 23:07:15.317065 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:15.317047    4431 scope.go:122] "RemoveContainer" containerID="d71101cf7f8cac4b13219889ee6bdde6099376f3a2df441098177a241ab2e114"
Nov 03 23:07:15.317361 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:07:15.317341    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d71101cf7f8cac4b13219889ee6bdde6099376f3a2df441098177a241ab2e114\": container with ID starting with d71101cf7f8cac4b13219889ee6bdde6099376f3a2df441098177a241ab2e114 not found: ID does not exist" containerID="d71101cf7f8cac4b13219889ee6bdde6099376f3a2df441098177a241ab2e114"
Nov 03 23:07:15.317419 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:15.317368    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d71101cf7f8cac4b13219889ee6bdde6099376f3a2df441098177a241ab2e114"} err="failed to get container status \"d71101cf7f8cac4b13219889ee6bdde6099376f3a2df441098177a241ab2e114\": rpc error: code = NotFound desc = could not find container \"d71101cf7f8cac4b13219889ee6bdde6099376f3a2df441098177a241ab2e114\": container with ID starting with d71101cf7f8cac4b13219889ee6bdde6099376f3a2df441098177a241ab2e114 not found: ID does not exist"
Nov 03 23:07:15.350607 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:15.350570    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-88a4d9/simulator-w0-88a4d9-kserve-785f84888f-sjbzp"]
Nov 03 23:07:15.357486 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:15.357460    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-88a4d9/simulator-w0-88a4d9-kserve-785f84888f-sjbzp"]
Nov 03 23:07:15.391678 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:15.391648    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-88a4d9/premium-w0-88a4d9-kserve-5559d75b7b-vqllq"]
Nov 03 23:07:15.397043 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:15.397021    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-88a4d9/premium-w0-88a4d9-kserve-5559d75b7b-vqllq"]
Nov 03 23:07:16.088096 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:16.088064    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05206737-4ca3-41fc-b35a-37c625cc182c" path="/var/lib/kubelet/pods/05206737-4ca3-41fc-b35a-37c625cc182c/volumes"
Nov 03 23:07:16.088504 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:16.088484    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18bd460b-a0f1-415c-bcb4-41c9f70d2926" path="/var/lib/kubelet/pods/18bd460b-a0f1-415c-bcb4-41c9f70d2926/volumes"
Nov 03 23:07:16.805302 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:16.805270    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:07:18.177895 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:18.177858    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-5187e1/distinct-w3-5187e1-kserve-545b7fb6bc-mk49g"]
Nov 03 23:07:18.189679 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:18.189650    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-5187e1/premium-w3-5187e1-kserve-56bf8788d4-pp8st"]
Nov 03 23:07:18.318463 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:18.318410    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:07:18.462706 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:18.462624    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:07:19.917000 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:19.916967    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:07:21.217647 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:21.217604    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d8a3d1-85f887b8c4-gfs5q" podUID="94198afd-2296-4658-ba19-6624a26f2bf9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 23:07:21.447353 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:21.447320    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:07:22.925079 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:22.925046    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:07:24.006365 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:24.006335    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:07:24.160895 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:24.160854    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:07:24.199013 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:24.198984    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d8a3d1-85f887b8c4-gfs5q"
Nov 03 23:07:24.229873 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:24.229753    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-88a4d9-6bb4875fbc-xrztv"]
Nov 03 23:07:24.230072 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:24.229975    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-88a4d9-6bb4875fbc-xrztv" podUID="11f1ee81-e4ca-4096-bf74-62018777d965" containerName="maas-api" containerID="cri-o://667b577c3c922122f3f20ee6d8cbbcde6396484ee5cb9c0ae72518b4ea47b3ce" gracePeriod=30
Nov 03 23:07:24.266885 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:24.266857    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94198afd-2296-4658-ba19-6624a26f2bf9-kube-api-access-zqw9v\" (UniqueName: \"kubernetes.io/projected/94198afd-2296-4658-ba19-6624a26f2bf9-kube-api-access-zqw9v\") pod \"94198afd-2296-4658-ba19-6624a26f2bf9\" (UID: \"94198afd-2296-4658-ba19-6624a26f2bf9\") "
Nov 03 23:07:24.267053 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:24.266997    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94198afd-2296-4658-ba19-6624a26f2bf9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94198afd-2296-4658-ba19-6624a26f2bf9-plugins-config-volume\") pod \"94198afd-2296-4658-ba19-6624a26f2bf9\" (UID: \"94198afd-2296-4658-ba19-6624a26f2bf9\") "
Nov 03 23:07:24.267447 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:24.267422    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94198afd-2296-4658-ba19-6624a26f2bf9-plugins-config-volume" pod "94198afd-2296-4658-ba19-6624a26f2bf9" (UID: "94198afd-2296-4658-ba19-6624a26f2bf9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:07:24.269033 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:24.269011    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94198afd-2296-4658-ba19-6624a26f2bf9-kube-api-access-zqw9v" pod "94198afd-2296-4658-ba19-6624a26f2bf9" (UID: "94198afd-2296-4658-ba19-6624a26f2bf9"). InnerVolumeSpecName "kube-api-access-zqw9v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:07:24.334609 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:24.334565    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:07:24.347523 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:24.347488    4431 generic.go:356] "Generic (PLEG): container finished" podID="11f1ee81-e4ca-4096-bf74-62018777d965" containerID="667b577c3c922122f3f20ee6d8cbbcde6396484ee5cb9c0ae72518b4ea47b3ce" exitCode=0
Nov 03 23:07:24.347676 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:24.347557    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-88a4d9-6bb4875fbc-xrztv" event={"ID":"11f1ee81-e4ca-4096-bf74-62018777d965","Type":"ContainerDied","Data":"667b577c3c922122f3f20ee6d8cbbcde6396484ee5cb9c0ae72518b4ea47b3ce"}
Nov 03 23:07:24.349137 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:24.349112    4431 generic.go:356] "Generic (PLEG): container finished" podID="94198afd-2296-4658-ba19-6624a26f2bf9" containerID="d3109ce13320cddcfdadb8985aa6bb60d472eb6cf56ed093b5485bb2799c820c" exitCode=0
Nov 03 23:07:24.349272 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:24.349176    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d8a3d1-85f887b8c4-gfs5q"
Nov 03 23:07:24.349272 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:24.349188    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d8a3d1-85f887b8c4-gfs5q" event={"ID":"94198afd-2296-4658-ba19-6624a26f2bf9","Type":"ContainerDied","Data":"d3109ce13320cddcfdadb8985aa6bb60d472eb6cf56ed093b5485bb2799c820c"}
Nov 03 23:07:24.349272 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:24.349249    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d8a3d1-85f887b8c4-gfs5q" event={"ID":"94198afd-2296-4658-ba19-6624a26f2bf9","Type":"ContainerDied","Data":"4e5d86cafbf9719ba74d778496bf3249f9dc4d459a7464f4ee7f4f6f1de91140"}
Nov 03 23:07:24.349640 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:24.349283    4431 scope.go:122] "RemoveContainer" containerID="d3109ce13320cddcfdadb8985aa6bb60d472eb6cf56ed093b5485bb2799c820c"
Nov 03 23:07:24.361898 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:24.361869    4431 scope.go:122] "RemoveContainer" containerID="d3109ce13320cddcfdadb8985aa6bb60d472eb6cf56ed093b5485bb2799c820c"
Nov 03 23:07:24.362272 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:07:24.362248    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3109ce13320cddcfdadb8985aa6bb60d472eb6cf56ed093b5485bb2799c820c\": container with ID starting with d3109ce13320cddcfdadb8985aa6bb60d472eb6cf56ed093b5485bb2799c820c not found: ID does not exist" containerID="d3109ce13320cddcfdadb8985aa6bb60d472eb6cf56ed093b5485bb2799c820c"
Nov 03 23:07:24.362389 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:24.362280    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3109ce13320cddcfdadb8985aa6bb60d472eb6cf56ed093b5485bb2799c820c"} err="failed to get container status \"d3109ce13320cddcfdadb8985aa6bb60d472eb6cf56ed093b5485bb2799c820c\": rpc error: code = NotFound desc = could not find container \"d3109ce13320cddcfdadb8985aa6bb60d472eb6cf56ed093b5485bb2799c820c\": container with ID starting with d3109ce13320cddcfdadb8985aa6bb60d472eb6cf56ed093b5485bb2799c820c not found: ID does not exist"
Nov 03 23:07:24.368062 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:24.368033    4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94198afd-2296-4658-ba19-6624a26f2bf9-plugins-config-volume\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:07:24.368197 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:24.368065    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqw9v\" (UniqueName: \"kubernetes.io/projected/94198afd-2296-4658-ba19-6624a26f2bf9-kube-api-access-zqw9v\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:07:24.400208 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:24.400173    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-d8a3d1-85f887b8c4-gfs5q"]
Nov 03 23:07:24.404370 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:24.404240    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-d8a3d1-85f887b8c4-gfs5q"]
Nov 03 23:07:24.485079 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:24.485013    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-88a4d9-6bb4875fbc-xrztv"
Nov 03 23:07:24.569475 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:24.569441    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11f1ee81-e4ca-4096-bf74-62018777d965-kube-api-access-qkgj7\" (UniqueName: \"kubernetes.io/projected/11f1ee81-e4ca-4096-bf74-62018777d965-kube-api-access-qkgj7\") pod \"11f1ee81-e4ca-4096-bf74-62018777d965\" (UID: \"11f1ee81-e4ca-4096-bf74-62018777d965\") "
Nov 03 23:07:24.569679 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:24.569494    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11f1ee81-e4ca-4096-bf74-62018777d965-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11f1ee81-e4ca-4096-bf74-62018777d965-openshift-service-ca\") pod \"11f1ee81-e4ca-4096-bf74-62018777d965\" (UID: \"11f1ee81-e4ca-4096-bf74-62018777d965\") "
Nov 03 23:07:24.569679 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:24.569527    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11f1ee81-e4ca-4096-bf74-62018777d965-metrics-certs\" (UniqueName: \"kubernetes.io/secret/11f1ee81-e4ca-4096-bf74-62018777d965-metrics-certs\") pod \"11f1ee81-e4ca-4096-bf74-62018777d965\" (UID: \"11f1ee81-e4ca-4096-bf74-62018777d965\") "
Nov 03 23:07:24.569679 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:24.569587    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11f1ee81-e4ca-4096-bf74-62018777d965-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11f1ee81-e4ca-4096-bf74-62018777d965-maas-api-tls\") pod \"11f1ee81-e4ca-4096-bf74-62018777d965\" (UID: \"11f1ee81-e4ca-4096-bf74-62018777d965\") "
Nov 03 23:07:24.569921 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:24.569890    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11f1ee81-e4ca-4096-bf74-62018777d965-openshift-service-ca" pod "11f1ee81-e4ca-4096-bf74-62018777d965" (UID: "11f1ee81-e4ca-4096-bf74-62018777d965"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:07:24.571666 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:24.571640    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11f1ee81-e4ca-4096-bf74-62018777d965-maas-api-tls" pod "11f1ee81-e4ca-4096-bf74-62018777d965" (UID: "11f1ee81-e4ca-4096-bf74-62018777d965"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:07:24.571784 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:24.571662    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11f1ee81-e4ca-4096-bf74-62018777d965-metrics-certs" pod "11f1ee81-e4ca-4096-bf74-62018777d965" (UID: "11f1ee81-e4ca-4096-bf74-62018777d965"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:07:24.571784 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:24.571682    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11f1ee81-e4ca-4096-bf74-62018777d965-kube-api-access-qkgj7" pod "11f1ee81-e4ca-4096-bf74-62018777d965" (UID: "11f1ee81-e4ca-4096-bf74-62018777d965"). InnerVolumeSpecName "kube-api-access-qkgj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:07:24.671184 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:24.671151    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkgj7\" (UniqueName: \"kubernetes.io/projected/11f1ee81-e4ca-4096-bf74-62018777d965-kube-api-access-qkgj7\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:07:24.671184 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:24.671177    4431 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11f1ee81-e4ca-4096-bf74-62018777d965-openshift-service-ca\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:07:24.671184 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:24.671186    4431 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11f1ee81-e4ca-4096-bf74-62018777d965-metrics-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:07:24.671443 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:24.671195    4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11f1ee81-e4ca-4096-bf74-62018777d965-maas-api-tls\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:07:24.807779 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:24.807703    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:07:24.960233 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:24.960181    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:07:25.357184 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:25.357142    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-88a4d9-6bb4875fbc-xrztv" event={"ID":"11f1ee81-e4ca-4096-bf74-62018777d965","Type":"ContainerDied","Data":"bda9f16f5baddd4053d2e36eaea5bed547982e02e5d13714f0ca78c3efbc0508"}
Nov 03 23:07:25.357648 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:25.357201    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-88a4d9-6bb4875fbc-xrztv"
Nov 03 23:07:25.357648 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:25.357267    4431 scope.go:122] "RemoveContainer" containerID="667b577c3c922122f3f20ee6d8cbbcde6396484ee5cb9c0ae72518b4ea47b3ce"
Nov 03 23:07:25.412793 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:25.412762    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-88a4d9-6bb4875fbc-xrztv"]
Nov 03 23:07:25.420585 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:25.420558    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:07:25.424053 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:25.424030    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-88a4d9-6bb4875fbc-xrztv"]
Nov 03 23:07:26.086385 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:26.086356    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11f1ee81-e4ca-4096-bf74-62018777d965" path="/var/lib/kubelet/pods/11f1ee81-e4ca-4096-bf74-62018777d965/volumes"
Nov 03 23:07:26.086718 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:26.086706    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94198afd-2296-4658-ba19-6624a26f2bf9" path="/var/lib/kubelet/pods/94198afd-2296-4658-ba19-6624a26f2bf9/volumes"
Nov 03 23:07:31.257770 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:07:31.257723    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 23:07:32.944095 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:32.944057    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:07:33.198070 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:33.197960    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-5187e1/distinct-w3-5187e1-kserve-545b7fb6bc-mk49g" podUID="3ba8dc17-1b1b-4811-9dd6-595d75c6ca04" containerName="main" containerID="cri-o://d9e2c5542308e597a66991f01ebfca061a9973a8ecae78e378eb21aa4f5e005e" gracePeriod=45
Nov 03 23:07:33.203122 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:33.203073    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:07:33.216165 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:33.215694    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-5187e1/premium-w3-5187e1-kserve-56bf8788d4-pp8st" podUID="a70ecfe9-020a-4331-b67c-aaaa46ed1eb1" containerName="main" containerID="cri-o://802fa2f1254e8db4a40a463b19e2778d32d1dcfa4dbc494eeb32c6872b05cede" gracePeriod=45
Nov 03 23:07:33.397755 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:33.393567    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:07:33.397755 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:33.397403    4431 generic.go:356] "Generic (PLEG): container finished" podID="3ba8dc17-1b1b-4811-9dd6-595d75c6ca04" containerID="d9e2c5542308e597a66991f01ebfca061a9973a8ecae78e378eb21aa4f5e005e" exitCode=0
Nov 03 23:07:33.397755 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:33.397519    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5187e1/distinct-w3-5187e1-kserve-545b7fb6bc-mk49g" event={"ID":"3ba8dc17-1b1b-4811-9dd6-595d75c6ca04","Type":"ContainerDied","Data":"d9e2c5542308e597a66991f01ebfca061a9973a8ecae78e378eb21aa4f5e005e"}
Nov 03 23:07:33.400404 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:33.399332    4431 generic.go:356] "Generic (PLEG): container finished" podID="a70ecfe9-020a-4331-b67c-aaaa46ed1eb1" containerID="802fa2f1254e8db4a40a463b19e2778d32d1dcfa4dbc494eeb32c6872b05cede" exitCode=0
Nov 03 23:07:33.400404 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:33.399403    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5187e1/premium-w3-5187e1-kserve-56bf8788d4-pp8st" event={"ID":"a70ecfe9-020a-4331-b67c-aaaa46ed1eb1","Type":"ContainerDied","Data":"802fa2f1254e8db4a40a463b19e2778d32d1dcfa4dbc494eeb32c6872b05cede"}
Nov 03 23:07:33.487687 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:33.487657    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5187e1/distinct-w3-5187e1-kserve-545b7fb6bc-mk49g"
Nov 03 23:07:33.515425 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:33.515401    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5187e1/premium-w3-5187e1-kserve-56bf8788d4-pp8st"
Nov 03 23:07:33.661942 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:33.661907    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a70ecfe9-020a-4331-b67c-aaaa46ed1eb1-home\" (UniqueName: \"kubernetes.io/empty-dir/a70ecfe9-020a-4331-b67c-aaaa46ed1eb1-home\") pod \"a70ecfe9-020a-4331-b67c-aaaa46ed1eb1\" (UID: \"a70ecfe9-020a-4331-b67c-aaaa46ed1eb1\") "
Nov 03 23:07:33.662133 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:33.661983    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a70ecfe9-020a-4331-b67c-aaaa46ed1eb1-tls-certs\" (UniqueName: \"kubernetes.io/secret/a70ecfe9-020a-4331-b67c-aaaa46ed1eb1-tls-certs\") pod \"a70ecfe9-020a-4331-b67c-aaaa46ed1eb1\" (UID: \"a70ecfe9-020a-4331-b67c-aaaa46ed1eb1\") "
Nov 03 23:07:33.662133 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:33.662032    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ba8dc17-1b1b-4811-9dd6-595d75c6ca04-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ba8dc17-1b1b-4811-9dd6-595d75c6ca04-dshm\") pod \"3ba8dc17-1b1b-4811-9dd6-595d75c6ca04\" (UID: \"3ba8dc17-1b1b-4811-9dd6-595d75c6ca04\") "
Nov 03 23:07:33.662133 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:33.662062    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ba8dc17-1b1b-4811-9dd6-595d75c6ca04-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ba8dc17-1b1b-4811-9dd6-595d75c6ca04-tmp-dir\") pod \"3ba8dc17-1b1b-4811-9dd6-595d75c6ca04\" (UID: \"3ba8dc17-1b1b-4811-9dd6-595d75c6ca04\") "
Nov 03 23:07:33.662133 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:33.662095    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a70ecfe9-020a-4331-b67c-aaaa46ed1eb1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a70ecfe9-020a-4331-b67c-aaaa46ed1eb1-dshm\") pod \"a70ecfe9-020a-4331-b67c-aaaa46ed1eb1\" (UID: \"a70ecfe9-020a-4331-b67c-aaaa46ed1eb1\") "
Nov 03 23:07:33.662133 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:33.662123    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ba8dc17-1b1b-4811-9dd6-595d75c6ca04-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ba8dc17-1b1b-4811-9dd6-595d75c6ca04-model-cache\") pod \"3ba8dc17-1b1b-4811-9dd6-595d75c6ca04\" (UID: \"3ba8dc17-1b1b-4811-9dd6-595d75c6ca04\") "
Nov 03 23:07:33.662443 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:33.662149    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ba8dc17-1b1b-4811-9dd6-595d75c6ca04-home\" (UniqueName: \"kubernetes.io/empty-dir/3ba8dc17-1b1b-4811-9dd6-595d75c6ca04-home\") pod \"3ba8dc17-1b1b-4811-9dd6-595d75c6ca04\" (UID: \"3ba8dc17-1b1b-4811-9dd6-595d75c6ca04\") "
Nov 03 23:07:33.662443 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:33.662191    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a70ecfe9-020a-4331-b67c-aaaa46ed1eb1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a70ecfe9-020a-4331-b67c-aaaa46ed1eb1-tmp-dir\") pod \"a70ecfe9-020a-4331-b67c-aaaa46ed1eb1\" (UID: \"a70ecfe9-020a-4331-b67c-aaaa46ed1eb1\") "
Nov 03 23:07:33.662443 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:33.662265    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ba8dc17-1b1b-4811-9dd6-595d75c6ca04-tls-certs\" (UniqueName: \"kubernetes.io/secret/3ba8dc17-1b1b-4811-9dd6-595d75c6ca04-tls-certs\") pod \"3ba8dc17-1b1b-4811-9dd6-595d75c6ca04\" (UID: \"3ba8dc17-1b1b-4811-9dd6-595d75c6ca04\") "
Nov 03 23:07:33.662443 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:33.662310    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a70ecfe9-020a-4331-b67c-aaaa46ed1eb1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a70ecfe9-020a-4331-b67c-aaaa46ed1eb1-model-cache\") pod \"a70ecfe9-020a-4331-b67c-aaaa46ed1eb1\" (UID: \"a70ecfe9-020a-4331-b67c-aaaa46ed1eb1\") "
Nov 03 23:07:33.662443 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:33.662371    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ba8dc17-1b1b-4811-9dd6-595d75c6ca04-kube-api-access-p5qbj\" (UniqueName: \"kubernetes.io/projected/3ba8dc17-1b1b-4811-9dd6-595d75c6ca04-kube-api-access-p5qbj\") pod \"3ba8dc17-1b1b-4811-9dd6-595d75c6ca04\" (UID: \"3ba8dc17-1b1b-4811-9dd6-595d75c6ca04\") "
Nov 03 23:07:33.662443 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:33.662402    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a70ecfe9-020a-4331-b67c-aaaa46ed1eb1-kube-api-access-xsfr4\" (UniqueName: \"kubernetes.io/projected/a70ecfe9-020a-4331-b67c-aaaa46ed1eb1-kube-api-access-xsfr4\") pod \"a70ecfe9-020a-4331-b67c-aaaa46ed1eb1\" (UID: \"a70ecfe9-020a-4331-b67c-aaaa46ed1eb1\") "
Nov 03 23:07:33.662777 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:33.662717    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ba8dc17-1b1b-4811-9dd6-595d75c6ca04-model-cache" pod "3ba8dc17-1b1b-4811-9dd6-595d75c6ca04" (UID: "3ba8dc17-1b1b-4811-9dd6-595d75c6ca04"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:07:33.663384 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:33.662877    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a70ecfe9-020a-4331-b67c-aaaa46ed1eb1-tmp-dir" pod "a70ecfe9-020a-4331-b67c-aaaa46ed1eb1" (UID: "a70ecfe9-020a-4331-b67c-aaaa46ed1eb1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:07:33.663384 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:33.662891    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a70ecfe9-020a-4331-b67c-aaaa46ed1eb1-model-cache" pod "a70ecfe9-020a-4331-b67c-aaaa46ed1eb1" (UID: "a70ecfe9-020a-4331-b67c-aaaa46ed1eb1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:07:33.663384 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:33.663285    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ba8dc17-1b1b-4811-9dd6-595d75c6ca04-tmp-dir" pod "3ba8dc17-1b1b-4811-9dd6-595d75c6ca04" (UID: "3ba8dc17-1b1b-4811-9dd6-595d75c6ca04"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:07:33.663384 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:33.663345    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ba8dc17-1b1b-4811-9dd6-595d75c6ca04-home" pod "3ba8dc17-1b1b-4811-9dd6-595d75c6ca04" (UID: "3ba8dc17-1b1b-4811-9dd6-595d75c6ca04"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:07:33.663831 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:33.663800    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a70ecfe9-020a-4331-b67c-aaaa46ed1eb1-home" pod "a70ecfe9-020a-4331-b67c-aaaa46ed1eb1" (UID: "a70ecfe9-020a-4331-b67c-aaaa46ed1eb1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:07:33.665394 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:33.665367    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a70ecfe9-020a-4331-b67c-aaaa46ed1eb1-kube-api-access-xsfr4" pod "a70ecfe9-020a-4331-b67c-aaaa46ed1eb1" (UID: "a70ecfe9-020a-4331-b67c-aaaa46ed1eb1"). InnerVolumeSpecName "kube-api-access-xsfr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:07:33.665502 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:33.665428    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ba8dc17-1b1b-4811-9dd6-595d75c6ca04-kube-api-access-p5qbj" pod "3ba8dc17-1b1b-4811-9dd6-595d75c6ca04" (UID: "3ba8dc17-1b1b-4811-9dd6-595d75c6ca04"). InnerVolumeSpecName "kube-api-access-p5qbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:07:33.665502 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:33.665489    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a70ecfe9-020a-4331-b67c-aaaa46ed1eb1-dshm" pod "a70ecfe9-020a-4331-b67c-aaaa46ed1eb1" (UID: "a70ecfe9-020a-4331-b67c-aaaa46ed1eb1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:07:33.666362 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:33.666330    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ba8dc17-1b1b-4811-9dd6-595d75c6ca04-tls-certs" pod "3ba8dc17-1b1b-4811-9dd6-595d75c6ca04" (UID: "3ba8dc17-1b1b-4811-9dd6-595d75c6ca04"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:07:33.669196 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:33.669165    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a70ecfe9-020a-4331-b67c-aaaa46ed1eb1-tls-certs" pod "a70ecfe9-020a-4331-b67c-aaaa46ed1eb1" (UID: "a70ecfe9-020a-4331-b67c-aaaa46ed1eb1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:07:33.671252 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:33.671042    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ba8dc17-1b1b-4811-9dd6-595d75c6ca04-dshm" pod "3ba8dc17-1b1b-4811-9dd6-595d75c6ca04" (UID: "3ba8dc17-1b1b-4811-9dd6-595d75c6ca04"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:07:33.764070 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:33.763973    4431 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a70ecfe9-020a-4331-b67c-aaaa46ed1eb1-dshm\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:07:33.764070 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:33.764006    4431 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ba8dc17-1b1b-4811-9dd6-595d75c6ca04-model-cache\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:07:33.764070 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:33.764028    4431 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ba8dc17-1b1b-4811-9dd6-595d75c6ca04-home\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:07:33.764070 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:33.764062    4431 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a70ecfe9-020a-4331-b67c-aaaa46ed1eb1-tmp-dir\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:07:33.764070 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:33.764071    4431 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ba8dc17-1b1b-4811-9dd6-595d75c6ca04-tls-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:07:33.764396 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:33.764079    4431 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a70ecfe9-020a-4331-b67c-aaaa46ed1eb1-model-cache\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:07:33.764396 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:33.764088    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5qbj\" (UniqueName: \"kubernetes.io/projected/3ba8dc17-1b1b-4811-9dd6-595d75c6ca04-kube-api-access-p5qbj\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:07:33.764396 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:33.764100    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsfr4\" (UniqueName: \"kubernetes.io/projected/a70ecfe9-020a-4331-b67c-aaaa46ed1eb1-kube-api-access-xsfr4\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:07:33.764396 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:33.764114    4431 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a70ecfe9-020a-4331-b67c-aaaa46ed1eb1-home\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:07:33.764396 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:33.764123    4431 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a70ecfe9-020a-4331-b67c-aaaa46ed1eb1-tls-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:07:33.764396 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:33.764131    4431 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ba8dc17-1b1b-4811-9dd6-595d75c6ca04-dshm\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:07:33.764396 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:33.764139    4431 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ba8dc17-1b1b-4811-9dd6-595d75c6ca04-tmp-dir\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:07:34.405561 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:34.405485    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5187e1/distinct-w3-5187e1-kserve-545b7fb6bc-mk49g"
Nov 03 23:07:34.405977 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:34.405490    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5187e1/distinct-w3-5187e1-kserve-545b7fb6bc-mk49g" event={"ID":"3ba8dc17-1b1b-4811-9dd6-595d75c6ca04","Type":"ContainerDied","Data":"259a55885b9da9a48c94bfe08702e73a9112980e01298ef511d51eb3aa9e3cde"}
Nov 03 23:07:34.405977 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:34.405603    4431 scope.go:122] "RemoveContainer" containerID="d9e2c5542308e597a66991f01ebfca061a9973a8ecae78e378eb21aa4f5e005e"
Nov 03 23:07:34.406926 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:34.406910    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5187e1/premium-w3-5187e1-kserve-56bf8788d4-pp8st"
Nov 03 23:07:34.406926 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:34.406912    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5187e1/premium-w3-5187e1-kserve-56bf8788d4-pp8st" event={"ID":"a70ecfe9-020a-4331-b67c-aaaa46ed1eb1","Type":"ContainerDied","Data":"ed2c6fd811d390d3c50d30f9e52f1c8d2ffcf20dab099c2a4422d835765328dd"}
Nov 03 23:07:34.414120 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:34.414102    4431 scope.go:122] "RemoveContainer" containerID="802fa2f1254e8db4a40a463b19e2778d32d1dcfa4dbc494eeb32c6872b05cede"
Nov 03 23:07:34.457710 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:34.457679    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-5187e1/distinct-w3-5187e1-kserve-545b7fb6bc-mk49g"]
Nov 03 23:07:34.469410 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:34.469384    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-5187e1/distinct-w3-5187e1-kserve-545b7fb6bc-mk49g"]
Nov 03 23:07:34.502263 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:34.502230    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-5187e1/premium-w3-5187e1-kserve-56bf8788d4-pp8st"]
Nov 03 23:07:34.506496 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:34.506473    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-5187e1/premium-w3-5187e1-kserve-56bf8788d4-pp8st"]
Nov 03 23:07:36.085798 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:36.085759    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ba8dc17-1b1b-4811-9dd6-595d75c6ca04" path="/var/lib/kubelet/pods/3ba8dc17-1b1b-4811-9dd6-595d75c6ca04/volumes"
Nov 03 23:07:36.086301 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:36.086284    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a70ecfe9-020a-4331-b67c-aaaa46ed1eb1" path="/var/lib/kubelet/pods/a70ecfe9-020a-4331-b67c-aaaa46ed1eb1/volumes"
Nov 03 23:07:36.409165 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:36.409081    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:07:36.648173 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:36.648141    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:07:38.629041 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:38.629003    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:07:44.377257 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:44.377162    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:07:44.523580 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:44.523545    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:07:44.795609 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:44.795567    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:07:50.986390 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:50.986342    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:07:55.330517 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:55.330484    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:07:55.606564 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:55.606479    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:07:55.784269 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:55.784207    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:07:56.002705 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:56.002668    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:07:56.146376 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:56.146341    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:07:56.581057 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:56.581017    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:07:56.948774 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:56.948749    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mx5q8_ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea/ovn-acl-logging/0.log"
Nov 03 23:07:56.958022 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:07:56.957999    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mx5q8_ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea/ovn-acl-logging/0.log"
Nov 03 23:08:01.258428 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:08:01.258389    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 23:08:02.304205 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:02.304172    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:08:16.234671 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:16.234634    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:08:18.978858 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:18.978823    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:08:19.116133 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:19.116095    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:08:23.003175 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.003083    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-w85lm"]
Nov 03 23:08:23.003746 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.003542    4431 cpu_manager.go:414] "containerMap: removing container" podUID="2452e573-42a1-40b4-b86f-3678286375bc" containerName="payload-processing"
Nov 03 23:08:23.003746 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.003554    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2452e573-42a1-40b4-b86f-3678286375bc" containerName="payload-processing"
Nov 03 23:08:23.003746 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.003569    4431 cpu_manager.go:414] "containerMap: removing container" podUID="a70ecfe9-020a-4331-b67c-aaaa46ed1eb1" containerName="main"
Nov 03 23:08:23.003746 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.003574    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a70ecfe9-020a-4331-b67c-aaaa46ed1eb1" containerName="main"
Nov 03 23:08:23.003746 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.003585    4431 cpu_manager.go:414] "containerMap: removing container" podUID="aa0f4c59-fbe8-4b5c-b2fb-452653ac95b9" containerName="payload-processing"
Nov 03 23:08:23.003746 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.003591    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa0f4c59-fbe8-4b5c-b2fb-452653ac95b9" containerName="payload-processing"
Nov 03 23:08:23.003746 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.003599    4431 cpu_manager.go:414] "containerMap: removing container" podUID="98293f18-47d0-49fe-8ec9-f6b80342985d" containerName="payload-pre-processing"
Nov 03 23:08:23.003746 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.003604    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98293f18-47d0-49fe-8ec9-f6b80342985d" containerName="payload-pre-processing"
Nov 03 23:08:23.003746 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.003612    4431 cpu_manager.go:414] "containerMap: removing container" podUID="0f78011a-db80-4c8f-b562-a80b6970d207" containerName="maas-api"
Nov 03 23:08:23.003746 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.003617    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f78011a-db80-4c8f-b562-a80b6970d207" containerName="maas-api"
Nov 03 23:08:23.003746 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.003624    4431 cpu_manager.go:414] "containerMap: removing container" podUID="c9745b64-3b13-46c2-bbd3-68730b829f6d" containerName="payload-processing"
Nov 03 23:08:23.003746 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.003630    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9745b64-3b13-46c2-bbd3-68730b829f6d" containerName="payload-processing"
Nov 03 23:08:23.003746 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.003641    4431 cpu_manager.go:414] "containerMap: removing container" podUID="c33b591b-ff1c-4385-99df-343b21eb04a8" containerName="istio-proxy"
Nov 03 23:08:23.003746 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.003646    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c33b591b-ff1c-4385-99df-343b21eb04a8" containerName="istio-proxy"
Nov 03 23:08:23.003746 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.003654    4431 cpu_manager.go:414] "containerMap: removing container" podUID="ab4a1779-440d-442f-bc77-751d89b544c7" containerName="maas-api"
Nov 03 23:08:23.003746 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.003658    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab4a1779-440d-442f-bc77-751d89b544c7" containerName="maas-api"
Nov 03 23:08:23.003746 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.003666    4431 cpu_manager.go:414] "containerMap: removing container" podUID="11f1ee81-e4ca-4096-bf74-62018777d965" containerName="maas-api"
Nov 03 23:08:23.003746 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.003670    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11f1ee81-e4ca-4096-bf74-62018777d965" containerName="maas-api"
Nov 03 23:08:23.003746 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.003676    4431 cpu_manager.go:414] "containerMap: removing container" podUID="94198afd-2296-4658-ba19-6624a26f2bf9" containerName="payload-processing"
Nov 03 23:08:23.003746 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.003680    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94198afd-2296-4658-ba19-6624a26f2bf9" containerName="payload-processing"
Nov 03 23:08:23.003746 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.003687    4431 cpu_manager.go:414] "containerMap: removing container" podUID="ffc24874-f9cc-4433-a4f4-2a72788b1067" containerName="istio-proxy"
Nov 03 23:08:23.003746 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.003691    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffc24874-f9cc-4433-a4f4-2a72788b1067" containerName="istio-proxy"
Nov 03 23:08:23.003746 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.003705    4431 cpu_manager.go:414] "containerMap: removing container" podUID="e45b1572-6519-4c71-884a-115eeddd86cf" containerName="main"
Nov 03 23:08:23.003746 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.003709    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e45b1572-6519-4c71-884a-115eeddd86cf" containerName="main"
Nov 03 23:08:23.003746 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.003716    4431 cpu_manager.go:414] "containerMap: removing container" podUID="44002d96-03c6-4063-b992-a531d4fe8016" containerName="istio-proxy"
Nov 03 23:08:23.003746 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.003720    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44002d96-03c6-4063-b992-a531d4fe8016" containerName="istio-proxy"
Nov 03 23:08:23.003746 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.003730    4431 cpu_manager.go:414] "containerMap: removing container" podUID="05206737-4ca3-41fc-b35a-37c625cc182c" containerName="main"
Nov 03 23:08:23.003746 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.003734    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05206737-4ca3-41fc-b35a-37c625cc182c" containerName="main"
Nov 03 23:08:23.003746 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.003742    4431 cpu_manager.go:414] "containerMap: removing container" podUID="0696d31c-b588-4817-ba3b-be9d9591eef9" containerName="maas-api"
Nov 03 23:08:23.003746 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.003746    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0696d31c-b588-4817-ba3b-be9d9591eef9" containerName="maas-api"
Nov 03 23:08:23.003746 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.003752    4431 cpu_manager.go:414] "containerMap: removing container" podUID="e90b239c-aeaa-4c94-97fb-6c2fd30f9244" containerName="istio-proxy"
Nov 03 23:08:23.003746 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.003757    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e90b239c-aeaa-4c94-97fb-6c2fd30f9244" containerName="istio-proxy"
Nov 03 23:08:23.003746 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.003765    4431 cpu_manager.go:414] "containerMap: removing container" podUID="fa0a680b-8f0d-4b19-8217-82282764511a" containerName="main"
Nov 03 23:08:23.003746 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.003769    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa0a680b-8f0d-4b19-8217-82282764511a" containerName="main"
Nov 03 23:08:23.005360 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.003775    4431 cpu_manager.go:414] "containerMap: removing container" podUID="18bd460b-a0f1-415c-bcb4-41c9f70d2926" containerName="main"
Nov 03 23:08:23.005360 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.003780    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18bd460b-a0f1-415c-bcb4-41c9f70d2926" containerName="main"
Nov 03 23:08:23.005360 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.003787    4431 cpu_manager.go:414] "containerMap: removing container" podUID="3ba8dc17-1b1b-4811-9dd6-595d75c6ca04" containerName="main"
Nov 03 23:08:23.005360 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.003791    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ba8dc17-1b1b-4811-9dd6-595d75c6ca04" containerName="main"
Nov 03 23:08:23.005360 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.003800    4431 cpu_manager.go:414] "containerMap: removing container" podUID="90310d72-c818-4fdd-b4e7-b51f633caa05" containerName="main"
Nov 03 23:08:23.005360 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.003804    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90310d72-c818-4fdd-b4e7-b51f633caa05" containerName="main"
Nov 03 23:08:23.005360 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.003810    4431 cpu_manager.go:414] "containerMap: removing container" podUID="ce50af62-9cb6-4379-99c7-b40630538f4f" containerName="main"
Nov 03 23:08:23.005360 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.003814    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce50af62-9cb6-4379-99c7-b40630538f4f" containerName="main"
Nov 03 23:08:23.005360 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.003820    4431 cpu_manager.go:414] "containerMap: removing container" podUID="db62e992-8f6d-4cba-b260-02bf3c06e839" containerName="istio-proxy"
Nov 03 23:08:23.005360 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.003824    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db62e992-8f6d-4cba-b260-02bf3c06e839" containerName="istio-proxy"
Nov 03 23:08:23.005360 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.003891    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="c33b591b-ff1c-4385-99df-343b21eb04a8" containerName="istio-proxy"
Nov 03 23:08:23.005360 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.003900    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab4a1779-440d-442f-bc77-751d89b544c7" containerName="maas-api"
Nov 03 23:08:23.005360 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.003906    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="11f1ee81-e4ca-4096-bf74-62018777d965" containerName="maas-api"
Nov 03 23:08:23.005360 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.003911    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="94198afd-2296-4658-ba19-6624a26f2bf9" containerName="payload-processing"
Nov 03 23:08:23.005360 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.003917    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffc24874-f9cc-4433-a4f4-2a72788b1067" containerName="istio-proxy"
Nov 03 23:08:23.005360 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.003925    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="e45b1572-6519-4c71-884a-115eeddd86cf" containerName="main"
Nov 03 23:08:23.005360 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.003930    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="44002d96-03c6-4063-b992-a531d4fe8016" containerName="istio-proxy"
Nov 03 23:08:23.005360 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.003936    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="05206737-4ca3-41fc-b35a-37c625cc182c" containerName="main"
Nov 03 23:08:23.005360 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.003942    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="0696d31c-b588-4817-ba3b-be9d9591eef9" containerName="maas-api"
Nov 03 23:08:23.005360 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.003946    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="e90b239c-aeaa-4c94-97fb-6c2fd30f9244" containerName="istio-proxy"
Nov 03 23:08:23.005360 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.003952    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa0a680b-8f0d-4b19-8217-82282764511a" containerName="main"
Nov 03 23:08:23.005360 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.003956    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="18bd460b-a0f1-415c-bcb4-41c9f70d2926" containerName="main"
Nov 03 23:08:23.005360 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.003961    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ba8dc17-1b1b-4811-9dd6-595d75c6ca04" containerName="main"
Nov 03 23:08:23.005360 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.003965    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="90310d72-c818-4fdd-b4e7-b51f633caa05" containerName="main"
Nov 03 23:08:23.005360 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.003970    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce50af62-9cb6-4379-99c7-b40630538f4f" containerName="main"
Nov 03 23:08:23.005360 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.003976    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="2452e573-42a1-40b4-b86f-3678286375bc" containerName="payload-processing"
Nov 03 23:08:23.005360 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.003981    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="a70ecfe9-020a-4331-b67c-aaaa46ed1eb1" containerName="main"
Nov 03 23:08:23.005360 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.003986    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa0f4c59-fbe8-4b5c-b2fb-452653ac95b9" containerName="payload-processing"
Nov 03 23:08:23.005360 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.003991    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="98293f18-47d0-49fe-8ec9-f6b80342985d" containerName="payload-pre-processing"
Nov 03 23:08:23.005360 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.003996    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f78011a-db80-4c8f-b562-a80b6970d207" containerName="maas-api"
Nov 03 23:08:23.005360 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.004001    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9745b64-3b13-46c2-bbd3-68730b829f6d" containerName="payload-processing"
Nov 03 23:08:23.007139 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.007118    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-w85lm"
Nov 03 23:08:23.019774 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.019749    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fxflc"]
Nov 03 23:08:23.022946 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.022930    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fxflc"
Nov 03 23:08:23.030264 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.030243    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-9d7jc\""
Nov 03 23:08:23.030768 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.030749    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 03 23:08:23.033042 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.033016    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-w85lm"]
Nov 03 23:08:23.044065 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.044039    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fxflc"]
Nov 03 23:08:23.112052 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.112015    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f80ff4c3-aff2-4ea7-89f8-ae103660fdc9-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-w85lm\" (UID: \"f80ff4c3-aff2-4ea7-89f8-ae103660fdc9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-w85lm"
Nov 03 23:08:23.112242 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.112060    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b051f82e-d473-415b-bba2-a659b94a106f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-fxflc\" (UID: \"b051f82e-d473-415b-bba2-a659b94a106f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fxflc"
Nov 03 23:08:23.112242 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.112145    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f80ff4c3-aff2-4ea7-89f8-ae103660fdc9-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-w85lm\" (UID: \"f80ff4c3-aff2-4ea7-89f8-ae103660fdc9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-w85lm"
Nov 03 23:08:23.112242 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.112181    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b051f82e-d473-415b-bba2-a659b94a106f-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-fxflc\" (UID: \"b051f82e-d473-415b-bba2-a659b94a106f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fxflc"
Nov 03 23:08:23.112242 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.112201    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jghr9\" (UniqueName: \"kubernetes.io/projected/f80ff4c3-aff2-4ea7-89f8-ae103660fdc9-kube-api-access-jghr9\") pod \"maas-default-gateway-openshift-default-7df854bf67-w85lm\" (UID: \"f80ff4c3-aff2-4ea7-89f8-ae103660fdc9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-w85lm"
Nov 03 23:08:23.112242 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.112236    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f80ff4c3-aff2-4ea7-89f8-ae103660fdc9-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-w85lm\" (UID: \"f80ff4c3-aff2-4ea7-89f8-ae103660fdc9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-w85lm"
Nov 03 23:08:23.112469 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.112295    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b051f82e-d473-415b-bba2-a659b94a106f-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-fxflc\" (UID: \"b051f82e-d473-415b-bba2-a659b94a106f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fxflc"
Nov 03 23:08:23.112469 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.112380    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b051f82e-d473-415b-bba2-a659b94a106f-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-fxflc\" (UID: \"b051f82e-d473-415b-bba2-a659b94a106f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fxflc"
Nov 03 23:08:23.112469 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.112430    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f80ff4c3-aff2-4ea7-89f8-ae103660fdc9-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-w85lm\" (UID: \"f80ff4c3-aff2-4ea7-89f8-ae103660fdc9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-w85lm"
Nov 03 23:08:23.112639 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.112531    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b051f82e-d473-415b-bba2-a659b94a106f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-fxflc\" (UID: \"b051f82e-d473-415b-bba2-a659b94a106f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fxflc"
Nov 03 23:08:23.112639 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.112570    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhf9z\" (UniqueName: \"kubernetes.io/projected/b051f82e-d473-415b-bba2-a659b94a106f-kube-api-access-mhf9z\") pod \"maas-default-gateway-openshift-default-7df854bf67-fxflc\" (UID: \"b051f82e-d473-415b-bba2-a659b94a106f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fxflc"
Nov 03 23:08:23.112761 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.112641    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b051f82e-d473-415b-bba2-a659b94a106f-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-fxflc\" (UID: \"b051f82e-d473-415b-bba2-a659b94a106f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fxflc"
Nov 03 23:08:23.112761 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.112692    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f80ff4c3-aff2-4ea7-89f8-ae103660fdc9-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-w85lm\" (UID: \"f80ff4c3-aff2-4ea7-89f8-ae103660fdc9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-w85lm"
Nov 03 23:08:23.112761 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.112752    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f80ff4c3-aff2-4ea7-89f8-ae103660fdc9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-w85lm\" (UID: \"f80ff4c3-aff2-4ea7-89f8-ae103660fdc9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-w85lm"
Nov 03 23:08:23.112932 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.112814    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f80ff4c3-aff2-4ea7-89f8-ae103660fdc9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-w85lm\" (UID: \"f80ff4c3-aff2-4ea7-89f8-ae103660fdc9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-w85lm"
Nov 03 23:08:23.112932 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.112838    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b051f82e-d473-415b-bba2-a659b94a106f-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-fxflc\" (UID: \"b051f82e-d473-415b-bba2-a659b94a106f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fxflc"
Nov 03 23:08:23.112932 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.112874    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b051f82e-d473-415b-bba2-a659b94a106f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-fxflc\" (UID: \"b051f82e-d473-415b-bba2-a659b94a106f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fxflc"
Nov 03 23:08:23.113086 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.112937    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f80ff4c3-aff2-4ea7-89f8-ae103660fdc9-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-w85lm\" (UID: \"f80ff4c3-aff2-4ea7-89f8-ae103660fdc9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-w85lm"
Nov 03 23:08:23.213901 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.213861    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b051f82e-d473-415b-bba2-a659b94a106f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-fxflc\" (UID: \"b051f82e-d473-415b-bba2-a659b94a106f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fxflc"
Nov 03 23:08:23.213901 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.213902    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhf9z\" (UniqueName: \"kubernetes.io/projected/b051f82e-d473-415b-bba2-a659b94a106f-kube-api-access-mhf9z\") pod \"maas-default-gateway-openshift-default-7df854bf67-fxflc\" (UID: \"b051f82e-d473-415b-bba2-a659b94a106f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fxflc"
Nov 03 23:08:23.214164 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.214019    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b051f82e-d473-415b-bba2-a659b94a106f-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-fxflc\" (UID: \"b051f82e-d473-415b-bba2-a659b94a106f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fxflc"
Nov 03 23:08:23.214164 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.214070    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f80ff4c3-aff2-4ea7-89f8-ae103660fdc9-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-w85lm\" (UID: \"f80ff4c3-aff2-4ea7-89f8-ae103660fdc9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-w85lm"
Nov 03 23:08:23.214164 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.214091    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f80ff4c3-aff2-4ea7-89f8-ae103660fdc9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-w85lm\" (UID: \"f80ff4c3-aff2-4ea7-89f8-ae103660fdc9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-w85lm"
Nov 03 23:08:23.214164 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.214136    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f80ff4c3-aff2-4ea7-89f8-ae103660fdc9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-w85lm\" (UID: \"f80ff4c3-aff2-4ea7-89f8-ae103660fdc9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-w85lm"
Nov 03 23:08:23.214164 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.214159    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b051f82e-d473-415b-bba2-a659b94a106f-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-fxflc\" (UID: \"b051f82e-d473-415b-bba2-a659b94a106f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fxflc"
Nov 03 23:08:23.214489 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.214195    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b051f82e-d473-415b-bba2-a659b94a106f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-fxflc\" (UID: \"b051f82e-d473-415b-bba2-a659b94a106f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fxflc"
Nov 03 23:08:23.214489 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.214286    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f80ff4c3-aff2-4ea7-89f8-ae103660fdc9-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-w85lm\" (UID: \"f80ff4c3-aff2-4ea7-89f8-ae103660fdc9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-w85lm"
Nov 03 23:08:23.214489 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.214366    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f80ff4c3-aff2-4ea7-89f8-ae103660fdc9-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-w85lm\" (UID: \"f80ff4c3-aff2-4ea7-89f8-ae103660fdc9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-w85lm"
Nov 03 23:08:23.214489 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.214414    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b051f82e-d473-415b-bba2-a659b94a106f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-fxflc\" (UID: \"b051f82e-d473-415b-bba2-a659b94a106f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fxflc"
Nov 03 23:08:23.214489 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.214476    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f80ff4c3-aff2-4ea7-89f8-ae103660fdc9-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-w85lm\" (UID: \"f80ff4c3-aff2-4ea7-89f8-ae103660fdc9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-w85lm"
Nov 03 23:08:23.214763 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.214517    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f80ff4c3-aff2-4ea7-89f8-ae103660fdc9-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-w85lm\" (UID: \"f80ff4c3-aff2-4ea7-89f8-ae103660fdc9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-w85lm"
Nov 03 23:08:23.214763 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.214526    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b051f82e-d473-415b-bba2-a659b94a106f-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-fxflc\" (UID: \"b051f82e-d473-415b-bba2-a659b94a106f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fxflc"
Nov 03 23:08:23.214763 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.214552    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jghr9\" (UniqueName: \"kubernetes.io/projected/f80ff4c3-aff2-4ea7-89f8-ae103660fdc9-kube-api-access-jghr9\") pod \"maas-default-gateway-openshift-default-7df854bf67-w85lm\" (UID: \"f80ff4c3-aff2-4ea7-89f8-ae103660fdc9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-w85lm"
Nov 03 23:08:23.214763 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.214586    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f80ff4c3-aff2-4ea7-89f8-ae103660fdc9-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-w85lm\" (UID: \"f80ff4c3-aff2-4ea7-89f8-ae103660fdc9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-w85lm"
Nov 03 23:08:23.214763 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.214609    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b051f82e-d473-415b-bba2-a659b94a106f-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-fxflc\" (UID: \"b051f82e-d473-415b-bba2-a659b94a106f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fxflc"
Nov 03 23:08:23.214763 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.214638    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b051f82e-d473-415b-bba2-a659b94a106f-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-fxflc\" (UID: \"b051f82e-d473-415b-bba2-a659b94a106f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fxflc"
Nov 03 23:08:23.214763 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.214666    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f80ff4c3-aff2-4ea7-89f8-ae103660fdc9-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-w85lm\" (UID: \"f80ff4c3-aff2-4ea7-89f8-ae103660fdc9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-w85lm"
Nov 03 23:08:23.214763 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.214740    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f80ff4c3-aff2-4ea7-89f8-ae103660fdc9-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-w85lm\" (UID: \"f80ff4c3-aff2-4ea7-89f8-ae103660fdc9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-w85lm"
Nov 03 23:08:23.215208 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.214786    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b051f82e-d473-415b-bba2-a659b94a106f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-fxflc\" (UID: \"b051f82e-d473-415b-bba2-a659b94a106f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fxflc"
Nov 03 23:08:23.215208 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.214892    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b051f82e-d473-415b-bba2-a659b94a106f-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-fxflc\" (UID: \"b051f82e-d473-415b-bba2-a659b94a106f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fxflc"
Nov 03 23:08:23.215208 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.214973    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f80ff4c3-aff2-4ea7-89f8-ae103660fdc9-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-w85lm\" (UID: \"f80ff4c3-aff2-4ea7-89f8-ae103660fdc9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-w85lm"
Nov 03 23:08:23.215208 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.214973    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f80ff4c3-aff2-4ea7-89f8-ae103660fdc9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-w85lm\" (UID: \"f80ff4c3-aff2-4ea7-89f8-ae103660fdc9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-w85lm"
Nov 03 23:08:23.215208 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.215026    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f80ff4c3-aff2-4ea7-89f8-ae103660fdc9-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-w85lm\" (UID: \"f80ff4c3-aff2-4ea7-89f8-ae103660fdc9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-w85lm"
Nov 03 23:08:23.215514 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.215278    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b051f82e-d473-415b-bba2-a659b94a106f-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-fxflc\" (UID: \"b051f82e-d473-415b-bba2-a659b94a106f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fxflc"
Nov 03 23:08:23.215688 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.215662    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b051f82e-d473-415b-bba2-a659b94a106f-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-fxflc\" (UID: \"b051f82e-d473-415b-bba2-a659b94a106f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fxflc"
Nov 03 23:08:23.215874 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.215856    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b051f82e-d473-415b-bba2-a659b94a106f-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-fxflc\" (UID: \"b051f82e-d473-415b-bba2-a659b94a106f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fxflc"
Nov 03 23:08:23.217137 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.217107    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b051f82e-d473-415b-bba2-a659b94a106f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-fxflc\" (UID: \"b051f82e-d473-415b-bba2-a659b94a106f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fxflc"
Nov 03 23:08:23.217284 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.217241    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f80ff4c3-aff2-4ea7-89f8-ae103660fdc9-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-w85lm\" (UID: \"f80ff4c3-aff2-4ea7-89f8-ae103660fdc9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-w85lm"
Nov 03 23:08:23.217371 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.217354    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f80ff4c3-aff2-4ea7-89f8-ae103660fdc9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-w85lm\" (UID: \"f80ff4c3-aff2-4ea7-89f8-ae103660fdc9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-w85lm"
Nov 03 23:08:23.217371 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.217365    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b051f82e-d473-415b-bba2-a659b94a106f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-fxflc\" (UID: \"b051f82e-d473-415b-bba2-a659b94a106f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fxflc"
Nov 03 23:08:23.237257 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.237205    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f80ff4c3-aff2-4ea7-89f8-ae103660fdc9-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-w85lm\" (UID: \"f80ff4c3-aff2-4ea7-89f8-ae103660fdc9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-w85lm"
Nov 03 23:08:23.237466 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.237334    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b051f82e-d473-415b-bba2-a659b94a106f-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-fxflc\" (UID: \"b051f82e-d473-415b-bba2-a659b94a106f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fxflc"
Nov 03 23:08:23.237466 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.237442    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jghr9\" (UniqueName: \"kubernetes.io/projected/f80ff4c3-aff2-4ea7-89f8-ae103660fdc9-kube-api-access-jghr9\") pod \"maas-default-gateway-openshift-default-7df854bf67-w85lm\" (UID: \"f80ff4c3-aff2-4ea7-89f8-ae103660fdc9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-w85lm"
Nov 03 23:08:23.237568 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.237452    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhf9z\" (UniqueName: \"kubernetes.io/projected/b051f82e-d473-415b-bba2-a659b94a106f-kube-api-access-mhf9z\") pod \"maas-default-gateway-openshift-default-7df854bf67-fxflc\" (UID: \"b051f82e-d473-415b-bba2-a659b94a106f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fxflc"
Nov 03 23:08:23.320012 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.319916    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-w85lm"
Nov 03 23:08:23.335884 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.335853    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fxflc"
Nov 03 23:08:23.631535 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.631505    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-w85lm"]
Nov 03 23:08:23.633651 ip-10-0-139-73 kubenswrapper[4431]: W1103 23:08:23.633622    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf80ff4c3_aff2_4ea7_89f8_ae103660fdc9.slice/crio-53608e0dabed79f49a03b637e7bfd74a141bebcb168fe828c8b928c30867987d WatchSource:0}: Error finding container 53608e0dabed79f49a03b637e7bfd74a141bebcb168fe828c8b928c30867987d: Status 404 returned error can't find the container with id 53608e0dabed79f49a03b637e7bfd74a141bebcb168fe828c8b928c30867987d
Nov 03 23:08:23.635690 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.635656    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:08:23.635760 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.635719    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:08:23.635760 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.635748    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:08:23.671198 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.671171    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fxflc"]
Nov 03 23:08:23.687375 ip-10-0-139-73 kubenswrapper[4431]: W1103 23:08:23.687347    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb051f82e_d473_415b_bba2_a659b94a106f.slice/crio-48aa5f8e21c262e5a68430d011315dd0b0653c35d20fbd2084a048042a41863b WatchSource:0}: Error finding container 48aa5f8e21c262e5a68430d011315dd0b0653c35d20fbd2084a048042a41863b: Status 404 returned error can't find the container with id 48aa5f8e21c262e5a68430d011315dd0b0653c35d20fbd2084a048042a41863b
Nov 03 23:08:23.689933 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.689901    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:08:23.690012 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.689969    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:08:23.690012 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.690001    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:08:23.791030 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:23.790963    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:08:24.634100 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:24.634063    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fxflc" event={"ID":"b051f82e-d473-415b-bba2-a659b94a106f","Type":"ContainerStarted","Data":"e208e583e00e46e80b94ca5d41f9e719535dfa106ebb1c2540c7354b3dfe7951"}
Nov 03 23:08:24.634532 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:24.634118    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fxflc" event={"ID":"b051f82e-d473-415b-bba2-a659b94a106f","Type":"ContainerStarted","Data":"48aa5f8e21c262e5a68430d011315dd0b0653c35d20fbd2084a048042a41863b"}
Nov 03 23:08:24.636084 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:24.636062    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-w85lm" event={"ID":"f80ff4c3-aff2-4ea7-89f8-ae103660fdc9","Type":"ContainerStarted","Data":"b60c46ccfb53b38025f10405befa038e8f45b5cbb23b1dbfddcbdcc5206c52c2"}
Nov 03 23:08:24.636139 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:24.636099    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-w85lm" event={"ID":"f80ff4c3-aff2-4ea7-89f8-ae103660fdc9","Type":"ContainerStarted","Data":"53608e0dabed79f49a03b637e7bfd74a141bebcb168fe828c8b928c30867987d"}
Nov 03 23:08:24.714035 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:24.713975    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fxflc" podStartSLOduration=2.713959142 podStartE2EDuration="2.713959142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:08:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:08:24.700999431 +0000 UTC m=+3328.726855294" watchObservedRunningTime="2026-11-03 23:08:24.713959142 +0000 UTC m=+3328.739815004"
Nov 03 23:08:24.758238 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:24.757249    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-w85lm" podStartSLOduration=2.757229566 podStartE2EDuration="2.757229566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:08:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:08:24.75694699 +0000 UTC m=+3328.782802855" watchObservedRunningTime="2026-11-03 23:08:24.757229566 +0000 UTC m=+3328.783085426"
Nov 03 23:08:25.320929 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:25.320863    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-w85lm"
Nov 03 23:08:25.322953 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:25.322702    4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-w85lm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Nov 03 23:08:25.322953 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:25.322744    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-w85lm" podUID="f80ff4c3-aff2-4ea7-89f8-ae103660fdc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Nov 03 23:08:25.336948 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:25.336924    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fxflc"
Nov 03 23:08:25.338184 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:25.338161    4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-fxflc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Nov 03 23:08:25.338350 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:25.338245    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fxflc" podUID="b051f82e-d473-415b-bba2-a659b94a106f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Nov 03 23:08:25.728766 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:25.728722    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:08:26.321291 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:26.321256    4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-w85lm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Nov 03 23:08:26.321486 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:26.321314    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-w85lm" podUID="f80ff4c3-aff2-4ea7-89f8-ae103660fdc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Nov 03 23:08:26.336780 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:26.336751    4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-fxflc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Nov 03 23:08:26.336878 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:26.336802    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fxflc" podUID="b051f82e-d473-415b-bba2-a659b94a106f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Nov 03 23:08:27.320804 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:27.320765    4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-w85lm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Nov 03 23:08:27.321339 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:27.320820    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-w85lm" podUID="f80ff4c3-aff2-4ea7-89f8-ae103660fdc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Nov 03 23:08:27.336289 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:27.336258    4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-fxflc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Nov 03 23:08:27.336401 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:27.336306    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fxflc" podUID="b051f82e-d473-415b-bba2-a659b94a106f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Nov 03 23:08:28.320457 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:28.320427    4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-w85lm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Nov 03 23:08:28.320631 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:28.320476    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-w85lm" podUID="f80ff4c3-aff2-4ea7-89f8-ae103660fdc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Nov 03 23:08:28.336797 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:28.336769    4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-fxflc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Nov 03 23:08:28.337106 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:28.336818    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fxflc" podUID="b051f82e-d473-415b-bba2-a659b94a106f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Nov 03 23:08:29.320409 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:29.320368    4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-w85lm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Nov 03 23:08:29.320606 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:29.320426    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-w85lm" podUID="f80ff4c3-aff2-4ea7-89f8-ae103660fdc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Nov 03 23:08:29.336697 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:29.336668    4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-fxflc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Nov 03 23:08:29.336805 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:29.336715    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fxflc" podUID="b051f82e-d473-415b-bba2-a659b94a106f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Nov 03 23:08:30.320532 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:30.320488    4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-w85lm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Nov 03 23:08:30.320722 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:30.320556    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-w85lm" podUID="f80ff4c3-aff2-4ea7-89f8-ae103660fdc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Nov 03 23:08:30.337034 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:30.337004    4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-fxflc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Nov 03 23:08:30.337425 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:30.337059    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fxflc" podUID="b051f82e-d473-415b-bba2-a659b94a106f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Nov 03 23:08:31.272914 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:08:31.272868    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 23:08:31.950130 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:31.950092    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-w85lm"
Nov 03 23:08:31.950568 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:31.950506    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-w85lm"
Nov 03 23:08:31.950648 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:31.950621    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fxflc"
Nov 03 23:08:31.951013 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:31.950900    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fxflc"
Nov 03 23:08:31.951473 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:31.951453    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-w85lm"
Nov 03 23:08:31.951730 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:31.951715    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fxflc"
Nov 03 23:08:51.104468 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:51.104430    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:08:56.185082 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:08:56.185049    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:09:01.258371 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:09:01.258330    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 23:09:31.258757 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:09:31.258669    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 23:09:33.083673 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:33.083642    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:09:38.224297 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:38.224260    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gvf4z"]
Nov 03 23:09:38.227534 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:38.227514    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gvf4z"
Nov 03 23:09:38.252643 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:38.252612    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gvf4z"]
Nov 03 23:09:38.357460 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:38.357425    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01158a33-90af-45e2-bf92-00a165ae2ac3-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-gvf4z\" (UID: \"01158a33-90af-45e2-bf92-00a165ae2ac3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gvf4z"
Nov 03 23:09:38.357460 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:38.357462    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01158a33-90af-45e2-bf92-00a165ae2ac3-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-gvf4z\" (UID: \"01158a33-90af-45e2-bf92-00a165ae2ac3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gvf4z"
Nov 03 23:09:38.357678 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:38.357484    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01158a33-90af-45e2-bf92-00a165ae2ac3-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-gvf4z\" (UID: \"01158a33-90af-45e2-bf92-00a165ae2ac3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gvf4z"
Nov 03 23:09:38.357678 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:38.357565    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01158a33-90af-45e2-bf92-00a165ae2ac3-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-gvf4z\" (UID: \"01158a33-90af-45e2-bf92-00a165ae2ac3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gvf4z"
Nov 03 23:09:38.357678 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:38.357626    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01158a33-90af-45e2-bf92-00a165ae2ac3-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-gvf4z\" (UID: \"01158a33-90af-45e2-bf92-00a165ae2ac3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gvf4z"
Nov 03 23:09:38.357678 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:38.357651    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01158a33-90af-45e2-bf92-00a165ae2ac3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-gvf4z\" (UID: \"01158a33-90af-45e2-bf92-00a165ae2ac3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gvf4z"
Nov 03 23:09:38.357678 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:38.357674    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01158a33-90af-45e2-bf92-00a165ae2ac3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-gvf4z\" (UID: \"01158a33-90af-45e2-bf92-00a165ae2ac3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gvf4z"
Nov 03 23:09:38.357847 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:38.357772    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2hvc\" (UniqueName: \"kubernetes.io/projected/01158a33-90af-45e2-bf92-00a165ae2ac3-kube-api-access-b2hvc\") pod \"maas-default-gateway-openshift-default-7df854bf67-gvf4z\" (UID: \"01158a33-90af-45e2-bf92-00a165ae2ac3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gvf4z"
Nov 03 23:09:38.357847 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:38.357821    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01158a33-90af-45e2-bf92-00a165ae2ac3-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-gvf4z\" (UID: \"01158a33-90af-45e2-bf92-00a165ae2ac3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gvf4z"
Nov 03 23:09:38.458384 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:38.458344    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01158a33-90af-45e2-bf92-00a165ae2ac3-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-gvf4z\" (UID: \"01158a33-90af-45e2-bf92-00a165ae2ac3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gvf4z"
Nov 03 23:09:38.458384 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:38.458385    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01158a33-90af-45e2-bf92-00a165ae2ac3-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-gvf4z\" (UID: \"01158a33-90af-45e2-bf92-00a165ae2ac3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gvf4z"
Nov 03 23:09:38.458636 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:38.458407    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01158a33-90af-45e2-bf92-00a165ae2ac3-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-gvf4z\" (UID: \"01158a33-90af-45e2-bf92-00a165ae2ac3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gvf4z"
Nov 03 23:09:38.458636 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:38.458424    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01158a33-90af-45e2-bf92-00a165ae2ac3-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-gvf4z\" (UID: \"01158a33-90af-45e2-bf92-00a165ae2ac3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gvf4z"
Nov 03 23:09:38.458636 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:38.458466    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01158a33-90af-45e2-bf92-00a165ae2ac3-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-gvf4z\" (UID: \"01158a33-90af-45e2-bf92-00a165ae2ac3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gvf4z"
Nov 03 23:09:38.458636 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:38.458486    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01158a33-90af-45e2-bf92-00a165ae2ac3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-gvf4z\" (UID: \"01158a33-90af-45e2-bf92-00a165ae2ac3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gvf4z"
Nov 03 23:09:38.458636 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:38.458508    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01158a33-90af-45e2-bf92-00a165ae2ac3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-gvf4z\" (UID: \"01158a33-90af-45e2-bf92-00a165ae2ac3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gvf4z"
Nov 03 23:09:38.458636 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:38.458537    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2hvc\" (UniqueName: \"kubernetes.io/projected/01158a33-90af-45e2-bf92-00a165ae2ac3-kube-api-access-b2hvc\") pod \"maas-default-gateway-openshift-default-7df854bf67-gvf4z\" (UID: \"01158a33-90af-45e2-bf92-00a165ae2ac3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gvf4z"
Nov 03 23:09:38.458636 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:38.458582    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01158a33-90af-45e2-bf92-00a165ae2ac3-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-gvf4z\" (UID: \"01158a33-90af-45e2-bf92-00a165ae2ac3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gvf4z"
Nov 03 23:09:38.458999 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:38.458848    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01158a33-90af-45e2-bf92-00a165ae2ac3-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-gvf4z\" (UID: \"01158a33-90af-45e2-bf92-00a165ae2ac3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gvf4z"
Nov 03 23:09:38.458999 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:38.458979    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01158a33-90af-45e2-bf92-00a165ae2ac3-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-gvf4z\" (UID: \"01158a33-90af-45e2-bf92-00a165ae2ac3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gvf4z"
Nov 03 23:09:38.459115 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:38.458998    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01158a33-90af-45e2-bf92-00a165ae2ac3-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-gvf4z\" (UID: \"01158a33-90af-45e2-bf92-00a165ae2ac3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gvf4z"
Nov 03 23:09:38.459154 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:38.459111    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01158a33-90af-45e2-bf92-00a165ae2ac3-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-gvf4z\" (UID: \"01158a33-90af-45e2-bf92-00a165ae2ac3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gvf4z"
Nov 03 23:09:38.459486 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:38.459466    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01158a33-90af-45e2-bf92-00a165ae2ac3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-gvf4z\" (UID: \"01158a33-90af-45e2-bf92-00a165ae2ac3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gvf4z"
Nov 03 23:09:38.460843 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:38.460823    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01158a33-90af-45e2-bf92-00a165ae2ac3-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-gvf4z\" (UID: \"01158a33-90af-45e2-bf92-00a165ae2ac3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gvf4z"
Nov 03 23:09:38.460952 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:38.460935    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01158a33-90af-45e2-bf92-00a165ae2ac3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-gvf4z\" (UID: \"01158a33-90af-45e2-bf92-00a165ae2ac3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gvf4z"
Nov 03 23:09:38.481677 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:38.481611    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01158a33-90af-45e2-bf92-00a165ae2ac3-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-gvf4z\" (UID: \"01158a33-90af-45e2-bf92-00a165ae2ac3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gvf4z"
Nov 03 23:09:38.481802 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:38.481716    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2hvc\" (UniqueName: \"kubernetes.io/projected/01158a33-90af-45e2-bf92-00a165ae2ac3-kube-api-access-b2hvc\") pod \"maas-default-gateway-openshift-default-7df854bf67-gvf4z\" (UID: \"01158a33-90af-45e2-bf92-00a165ae2ac3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gvf4z"
Nov 03 23:09:38.542106 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:38.542075    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gvf4z"
Nov 03 23:09:38.857209 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:38.857110    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gvf4z"]
Nov 03 23:09:38.861243 ip-10-0-139-73 kubenswrapper[4431]: W1103 23:09:38.861204    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01158a33_90af_45e2_bf92_00a165ae2ac3.slice/crio-d6cec261727b56613c7a901b5df5257f40332c4e8ad1dfa2c53b8f96cf2ce72c WatchSource:0}: Error finding container d6cec261727b56613c7a901b5df5257f40332c4e8ad1dfa2c53b8f96cf2ce72c: Status 404 returned error can't find the container with id d6cec261727b56613c7a901b5df5257f40332c4e8ad1dfa2c53b8f96cf2ce72c
Nov 03 23:09:38.863411 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:38.863379    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:09:38.863480 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:38.863449    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:09:38.863522 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:38.863482    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:09:39.019471 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:39.019417    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gvf4z" event={"ID":"01158a33-90af-45e2-bf92-00a165ae2ac3","Type":"ContainerStarted","Data":"4b0d40c5a4542874eb3073bca9c04ef7fac7e577f96ebf1e223962be7d1bbaee"}
Nov 03 23:09:39.019673 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:39.019489    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gvf4z" event={"ID":"01158a33-90af-45e2-bf92-00a165ae2ac3","Type":"ContainerStarted","Data":"d6cec261727b56613c7a901b5df5257f40332c4e8ad1dfa2c53b8f96cf2ce72c"}
Nov 03 23:09:39.063676 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:39.063612    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gvf4z" podStartSLOduration=1.063593128 podStartE2EDuration="1.063593128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:09:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:09:39.062511908 +0000 UTC m=+3403.088367770" watchObservedRunningTime="2026-11-03 23:09:39.063593128 +0000 UTC m=+3403.089448991"
Nov 03 23:09:39.547157 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:39.547119    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gvf4z"
Nov 03 23:09:39.547899 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:39.547873    4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-gvf4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Nov 03 23:09:39.548087 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:39.548059    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gvf4z" podUID="01158a33-90af-45e2-bf92-00a165ae2ac3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Nov 03 23:09:40.554240 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:40.549652    4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-gvf4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Nov 03 23:09:40.554240 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:40.549728    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gvf4z" podUID="01158a33-90af-45e2-bf92-00a165ae2ac3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Nov 03 23:09:41.550240 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:41.543624    4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-gvf4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Nov 03 23:09:41.550240 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:41.543705    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gvf4z" podUID="01158a33-90af-45e2-bf92-00a165ae2ac3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Nov 03 23:09:42.561240 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:42.560177    4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-gvf4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Nov 03 23:09:42.561240 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:42.560270    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gvf4z" podUID="01158a33-90af-45e2-bf92-00a165ae2ac3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Nov 03 23:09:43.543037 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:43.542606    4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-gvf4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Nov 03 23:09:43.543037 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:43.542667    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gvf4z" podUID="01158a33-90af-45e2-bf92-00a165ae2ac3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Nov 03 23:09:44.549680 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:44.549641    4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-gvf4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Nov 03 23:09:44.550336 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:44.550300    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gvf4z" podUID="01158a33-90af-45e2-bf92-00a165ae2ac3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Nov 03 23:09:45.543141 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:45.543105    4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-gvf4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Nov 03 23:09:45.543387 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:45.543175    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gvf4z" podUID="01158a33-90af-45e2-bf92-00a165ae2ac3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Nov 03 23:09:47.544180 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:47.544127    4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-gvf4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 23:09:47.544711 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:47.544187    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gvf4z" podUID="01158a33-90af-45e2-bf92-00a165ae2ac3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:09:48.585435 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:48.585394    4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-gvf4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 23:09:48.585951 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:48.585454    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gvf4z" podUID="01158a33-90af-45e2-bf92-00a165ae2ac3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:09:49.053036 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:49.053005    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gvf4z"
Nov 03 23:09:49.053358 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:49.053330    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gvf4z"
Nov 03 23:09:49.054123 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:49.054105    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gvf4z"
Nov 03 23:09:51.230770 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:51.230730    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:09:53.260501 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:53.260433    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zfx74"]
Nov 03 23:09:53.264384 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:53.264362    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zfx74"
Nov 03 23:09:53.294183 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:53.294152    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zfx74"]
Nov 03 23:09:53.318856 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:53.318825    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cee6d2b-00f8-425f-a4ec-75d80d3844d9-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-zfx74\" (UID: \"1cee6d2b-00f8-425f-a4ec-75d80d3844d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zfx74"
Nov 03 23:09:53.318856 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:53.318858    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cee6d2b-00f8-425f-a4ec-75d80d3844d9-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-zfx74\" (UID: \"1cee6d2b-00f8-425f-a4ec-75d80d3844d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zfx74"
Nov 03 23:09:53.319091 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:53.318888    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cee6d2b-00f8-425f-a4ec-75d80d3844d9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-zfx74\" (UID: \"1cee6d2b-00f8-425f-a4ec-75d80d3844d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zfx74"
Nov 03 23:09:53.319091 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:53.318990    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cee6d2b-00f8-425f-a4ec-75d80d3844d9-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-zfx74\" (UID: \"1cee6d2b-00f8-425f-a4ec-75d80d3844d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zfx74"
Nov 03 23:09:53.319091 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:53.319036    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cee6d2b-00f8-425f-a4ec-75d80d3844d9-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-zfx74\" (UID: \"1cee6d2b-00f8-425f-a4ec-75d80d3844d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zfx74"
Nov 03 23:09:53.319091 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:53.319057    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5zjx\" (UniqueName: \"kubernetes.io/projected/1cee6d2b-00f8-425f-a4ec-75d80d3844d9-kube-api-access-w5zjx\") pod \"maas-default-gateway-openshift-default-7df854bf67-zfx74\" (UID: \"1cee6d2b-00f8-425f-a4ec-75d80d3844d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zfx74"
Nov 03 23:09:53.319357 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:53.319165    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cee6d2b-00f8-425f-a4ec-75d80d3844d9-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-zfx74\" (UID: \"1cee6d2b-00f8-425f-a4ec-75d80d3844d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zfx74"
Nov 03 23:09:53.319357 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:53.319207    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cee6d2b-00f8-425f-a4ec-75d80d3844d9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-zfx74\" (UID: \"1cee6d2b-00f8-425f-a4ec-75d80d3844d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zfx74"
Nov 03 23:09:53.319357 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:53.319295    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cee6d2b-00f8-425f-a4ec-75d80d3844d9-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-zfx74\" (UID: \"1cee6d2b-00f8-425f-a4ec-75d80d3844d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zfx74"
Nov 03 23:09:53.420075 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:53.420041    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cee6d2b-00f8-425f-a4ec-75d80d3844d9-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-zfx74\" (UID: \"1cee6d2b-00f8-425f-a4ec-75d80d3844d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zfx74"
Nov 03 23:09:53.420075 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:53.420077    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cee6d2b-00f8-425f-a4ec-75d80d3844d9-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-zfx74\" (UID: \"1cee6d2b-00f8-425f-a4ec-75d80d3844d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zfx74"
Nov 03 23:09:53.420370 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:53.420107    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cee6d2b-00f8-425f-a4ec-75d80d3844d9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-zfx74\" (UID: \"1cee6d2b-00f8-425f-a4ec-75d80d3844d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zfx74"
Nov 03 23:09:53.420370 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:53.420131    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cee6d2b-00f8-425f-a4ec-75d80d3844d9-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-zfx74\" (UID: \"1cee6d2b-00f8-425f-a4ec-75d80d3844d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zfx74"
Nov 03 23:09:53.420370 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:53.420160    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cee6d2b-00f8-425f-a4ec-75d80d3844d9-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-zfx74\" (UID: \"1cee6d2b-00f8-425f-a4ec-75d80d3844d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zfx74"
Nov 03 23:09:53.420370 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:53.420177    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5zjx\" (UniqueName: \"kubernetes.io/projected/1cee6d2b-00f8-425f-a4ec-75d80d3844d9-kube-api-access-w5zjx\") pod \"maas-default-gateway-openshift-default-7df854bf67-zfx74\" (UID: \"1cee6d2b-00f8-425f-a4ec-75d80d3844d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zfx74"
Nov 03 23:09:53.420370 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:53.420204    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cee6d2b-00f8-425f-a4ec-75d80d3844d9-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-zfx74\" (UID: \"1cee6d2b-00f8-425f-a4ec-75d80d3844d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zfx74"
Nov 03 23:09:53.420370 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:53.420247    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cee6d2b-00f8-425f-a4ec-75d80d3844d9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-zfx74\" (UID: \"1cee6d2b-00f8-425f-a4ec-75d80d3844d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zfx74"
Nov 03 23:09:53.420370 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:53.420297    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cee6d2b-00f8-425f-a4ec-75d80d3844d9-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-zfx74\" (UID: \"1cee6d2b-00f8-425f-a4ec-75d80d3844d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zfx74"
Nov 03 23:09:53.420728 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:53.420597    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cee6d2b-00f8-425f-a4ec-75d80d3844d9-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-zfx74\" (UID: \"1cee6d2b-00f8-425f-a4ec-75d80d3844d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zfx74"
Nov 03 23:09:53.420773 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:53.420734    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cee6d2b-00f8-425f-a4ec-75d80d3844d9-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-zfx74\" (UID: \"1cee6d2b-00f8-425f-a4ec-75d80d3844d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zfx74"
Nov 03 23:09:53.420810 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:53.420774    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cee6d2b-00f8-425f-a4ec-75d80d3844d9-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-zfx74\" (UID: \"1cee6d2b-00f8-425f-a4ec-75d80d3844d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zfx74"
Nov 03 23:09:53.420873 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:53.420853    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cee6d2b-00f8-425f-a4ec-75d80d3844d9-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-zfx74\" (UID: \"1cee6d2b-00f8-425f-a4ec-75d80d3844d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zfx74"
Nov 03 23:09:53.421210 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:53.421186    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cee6d2b-00f8-425f-a4ec-75d80d3844d9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-zfx74\" (UID: \"1cee6d2b-00f8-425f-a4ec-75d80d3844d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zfx74"
Nov 03 23:09:53.422694 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:53.422669    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cee6d2b-00f8-425f-a4ec-75d80d3844d9-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-zfx74\" (UID: \"1cee6d2b-00f8-425f-a4ec-75d80d3844d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zfx74"
Nov 03 23:09:53.422926 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:53.422906    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cee6d2b-00f8-425f-a4ec-75d80d3844d9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-zfx74\" (UID: \"1cee6d2b-00f8-425f-a4ec-75d80d3844d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zfx74"
Nov 03 23:09:53.443256 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:53.443203    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cee6d2b-00f8-425f-a4ec-75d80d3844d9-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-zfx74\" (UID: \"1cee6d2b-00f8-425f-a4ec-75d80d3844d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zfx74"
Nov 03 23:09:53.443403 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:53.443381    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5zjx\" (UniqueName: \"kubernetes.io/projected/1cee6d2b-00f8-425f-a4ec-75d80d3844d9-kube-api-access-w5zjx\") pod \"maas-default-gateway-openshift-default-7df854bf67-zfx74\" (UID: \"1cee6d2b-00f8-425f-a4ec-75d80d3844d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zfx74"
Nov 03 23:09:53.577772 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:53.577676    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zfx74"
Nov 03 23:09:53.891889 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:53.891755    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zfx74"]
Nov 03 23:09:53.896819 ip-10-0-139-73 kubenswrapper[4431]: W1103 23:09:53.896790    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cee6d2b_00f8_425f_a4ec_75d80d3844d9.slice/crio-3e30d3bdaee3294cd4f56c2904b1dd0d11b03724aba8a5a7503fd2319879b0b2 WatchSource:0}: Error finding container 3e30d3bdaee3294cd4f56c2904b1dd0d11b03724aba8a5a7503fd2319879b0b2: Status 404 returned error can't find the container with id 3e30d3bdaee3294cd4f56c2904b1dd0d11b03724aba8a5a7503fd2319879b0b2
Nov 03 23:09:53.898976 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:53.898939    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:09:53.899092 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:53.899021    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:09:53.899092 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:53.899083    4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 23:09:54.104170 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:54.104135    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zfx74" event={"ID":"1cee6d2b-00f8-425f-a4ec-75d80d3844d9","Type":"ContainerStarted","Data":"89119bf548d3e6ae1020a938e30f8559f3ba7939e8d5d3af315ce1743927a66d"}
Nov 03 23:09:54.104404 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:54.104186    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zfx74" event={"ID":"1cee6d2b-00f8-425f-a4ec-75d80d3844d9","Type":"ContainerStarted","Data":"3e30d3bdaee3294cd4f56c2904b1dd0d11b03724aba8a5a7503fd2319879b0b2"}
Nov 03 23:09:54.149145 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:54.149026    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zfx74" podStartSLOduration=1.149011309 podStartE2EDuration="1.149011309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:09:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:09:54.148115827 +0000 UTC m=+3418.173971689" watchObservedRunningTime="2026-11-03 23:09:54.149011309 +0000 UTC m=+3418.174867171"
Nov 03 23:09:54.578146 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:54.578110    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zfx74"
Nov 03 23:09:54.579743 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:54.579708    4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-zfx74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Nov 03 23:09:54.579890 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:54.579756    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zfx74" podUID="1cee6d2b-00f8-425f-a4ec-75d80d3844d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Nov 03 23:09:54.760237 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:54.759983    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:09:55.578769 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:55.578733    4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-zfx74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Nov 03 23:09:55.579156 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:55.578803    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zfx74" podUID="1cee6d2b-00f8-425f-a4ec-75d80d3844d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Nov 03 23:09:56.319182 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:56.319146    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:09:56.578740 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:56.578656    4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-zfx74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Nov 03 23:09:56.578740 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:56.578708    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zfx74" podUID="1cee6d2b-00f8-425f-a4ec-75d80d3844d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Nov 03 23:09:57.578636 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:57.578598    4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-zfx74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Nov 03 23:09:57.578809 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:57.578667    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zfx74" podUID="1cee6d2b-00f8-425f-a4ec-75d80d3844d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Nov 03 23:09:58.578737 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:58.578701    4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-zfx74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Nov 03 23:09:58.579144 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:58.578753    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zfx74" podUID="1cee6d2b-00f8-425f-a4ec-75d80d3844d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Nov 03 23:09:59.578572 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:59.578533    4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-zfx74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Nov 03 23:09:59.578784 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:09:59.578594    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zfx74" podUID="1cee6d2b-00f8-425f-a4ec-75d80d3844d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Nov 03 23:10:00.578630 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:10:00.578587    4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-zfx74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Nov 03 23:10:00.578929 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:10:00.578896    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zfx74" podUID="1cee6d2b-00f8-425f-a4ec-75d80d3844d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Nov 03 23:10:01.334981 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:10:01.334937    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 23:10:01.578917 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:10:01.578879    4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-zfx74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Nov 03 23:10:01.579575 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:10:01.579541    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zfx74" podUID="1cee6d2b-00f8-425f-a4ec-75d80d3844d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Nov 03 23:10:03.578678 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:10:03.578628    4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-zfx74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 23:10:03.579104 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:10:03.578719    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zfx74" podUID="1cee6d2b-00f8-425f-a4ec-75d80d3844d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:10:04.620396 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:10:04.620357    4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-zfx74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 23:10:04.620880 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:10:04.620421    4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zfx74" podUID="1cee6d2b-00f8-425f-a4ec-75d80d3844d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:10:04.948989 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:10:04.948958    4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zfx74"
Nov 03 23:10:04.949268 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:10:04.949245    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zfx74"
Nov 03 23:10:04.950084 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:10:04.950062    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zfx74"
Nov 03 23:10:31.257929 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:10:31.257893    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 23:10:51.347238 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:10:51.347177    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:10:56.423630 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:10:56.423583    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:11:01.257761 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:11:01.257717    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 23:11:31.258732 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:11:31.258691    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 23:11:51.464695 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:11:51.464657    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:11:56.544027 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:11:56.543983    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:12:01.258130 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:12:01.258093    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 23:12:31.258663 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:12:31.258622    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 23:12:51.574367 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:12:51.574330    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:12:56.618075 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:12:56.618040    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:12:56.978560 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:12:56.978530    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mx5q8_ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea/ovn-acl-logging/0.log"
Nov 03 23:12:56.989230 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:12:56.989193    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mx5q8_ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea/ovn-acl-logging/0.log"
Nov 03 23:13:01.328737 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:13:01.328698    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 23:13:12.854497 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:13:12.854460    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:13:31.362247 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:13:31.346410    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 23:13:39.010482 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:13:39.010444    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:13:39.497978 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:13:39.497935    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:13:39.660817 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:13:39.660780    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:13:43.494599 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:13:43.494563    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:13:43.641691 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:13:43.641658    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:13:44.981448 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:13:44.981411    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:13:45.133669 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:13:45.133637    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:13:45.282717 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:13:45.282624    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:13:46.810978 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:13:46.810944    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:13:48.415562 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:13:48.415530    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:13:50.180080 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:13:50.180041    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:13:51.715306 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:13:51.715265    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:13:51.853953 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:13:51.853816    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:13:53.547805 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:13:53.547771    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:13:55.224897 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:13:55.224840    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:13:56.780702 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:13:56.780670    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:13:56.920708 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:13:56.920665    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:13:58.336825 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:13:58.336786    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:13:58.525196 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:13:58.525163    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:13:58.925234 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:13:58.925188    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:13:59.071018 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:13:59.070982    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:13:59.214420 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:13:59.214342    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:14:00.225419 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:14:00.225373    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:14:00.379981 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:14:00.379944    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:14:01.258622 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:14:01.258586    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 23:14:02.005891 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:14:02.005857    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:14:03.714881 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:14:03.714842    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:14:05.521603 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:14:05.521568    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:14:07.207920 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:14:07.207885    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:14:08.806296 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:14:08.806261    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:14:10.423700 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:14:10.423659    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:14:12.180250 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:14:12.180173    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:14:13.810747 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:14:13.810711    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:14:15.410269 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:14:15.410234    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:14:17.146580 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:14:17.146532    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:14:18.838032 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:14:18.837999    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:14:20.545284 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:14:20.545233    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:14:22.208093 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:14:22.208053    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:14:23.906924 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:14:23.906887    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:14:25.513728 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:14:25.513694    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:14:27.207549 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:14:27.207508    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:14:28.799980 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:14:28.799945    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:14:30.512619 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:14:30.512589    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:14:31.257636 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:14:31.257603    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 23:14:32.206763 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:14:32.206730    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:14:33.992961 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:14:33.992917    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:14:35.626473 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:14:35.626431    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:14:37.212604 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:14:37.212563    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:14:38.913250 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:14:38.913195    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:14:40.526443 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:14:40.526405    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:14:42.208505 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:14:42.208459    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:14:43.706305 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:14:43.706268    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:14:43.845650 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:14:43.845590    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:14:45.763168 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:14:45.763130    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:14:47.505792 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:14:47.505753    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:14:48.726026 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:14:48.725980    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:14:49.125101 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:14:49.125018    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:14:50.707418 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:14:50.707387    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:14:51.827095 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:14:51.827058    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:14:52.407629 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:14:52.407574    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:14:52.897319 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:14:52.897281    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:14:54.004539 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:14:54.004499    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:14:54.495759 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:14:54.495719    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:14:54.633027 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:14:54.632989    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:14:55.493331 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:14:55.493295    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:14:55.959443 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:14:55.959402    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:14:56.103917 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:14:56.103885    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:14:56.603406 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:14:56.603372    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:14:56.902987 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:14:56.902899    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:15:01.257843 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:15:01.257807    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 23:15:02.652200 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:15:02.652161    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:15:02.794835 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:15:02.794794    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:15:03.008937 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:15:03.008902    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:15:25.385766 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:15:25.385675    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:15:26.061392 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:15:26.061333    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:15:31.259045 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:15:31.259008    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 23:15:31.784547 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:15:31.784512    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:15:43.412995 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:15:43.412954    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:15:43.557186 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:15:43.557150    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:15:51.951642 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:15:51.951607    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:15:53.018351 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:15:53.018311    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:15:57.027437 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:15:57.027399    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:16:01.257877 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:16:01.257838    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 23:16:31.258777 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:16:31.258732    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 23:16:52.145901 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:16:52.145862    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:16:53.138022 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:16:53.137983    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:16:57.156982 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:16:57.156940    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:17:01.258281 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:17:01.258244    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 23:17:21.331181 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:17:21.331145    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:17:21.468880 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:17:21.468841    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:17:26.203663 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:17:26.203622    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:17:26.663062 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:17:26.663023    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:17:26.934898 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:17:26.934797    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:17:30.607008 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:17:30.606959    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:17:30.762411 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:17:30.762357    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:17:31.258822 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:17:31.258782    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 23:17:32.195076 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:17:32.195040    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:17:32.346602 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:17:32.346564    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:17:32.566673 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:17:32.566569    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:17:34.099766 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:17:34.099728    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:17:35.923271 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:17:35.923233    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:17:37.602320 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:17:37.602276    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:17:39.216318 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:17:39.216287    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:17:40.851890 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:17:40.851851    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:17:40.998436 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:17:40.998403    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:17:42.624479 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:17:42.624444    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:17:44.205827 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:17:44.205796    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:17:45.478746 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:17:45.478713    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:17:46.022507 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:17:46.022469    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:17:46.170953 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:17:46.170920    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:17:46.398197 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:17:46.398117    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:17:47.575123 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:17:47.575090    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:17:47.915171 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:17:47.915140    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:17:49.625242 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:17:49.625177    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:17:51.310356 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:17:51.310305    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:17:52.265863 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:17:52.265835    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:17:53.006201 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:17:53.006172    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:17:54.554260 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:17:54.554173    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:17:54.721062 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:17:54.721027    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:17:56.310701 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:17:56.310668    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:17:56.802440 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:17:56.802403    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:17:57.007250 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:17:57.007196    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:17:57.012095 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:17:57.012071    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mx5q8_ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea/ovn-acl-logging/0.log"
Nov 03 23:17:57.022808 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:17:57.022785    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mx5q8_ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea/ovn-acl-logging/0.log"
Nov 03 23:17:57.338237 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:17:57.338182    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:17:58.010492 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:17:58.010453    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:17:59.609562 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:17:59.609524    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:18:01.204270 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:18:01.204232    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:18:01.258644 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:18:01.258604    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 23:18:02.817767 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:18:02.817727    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:18:04.501946 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:18:04.501908    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:18:06.228592 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:18:06.228557    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:18:08.013622 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:18:08.013587    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:18:09.618696 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:18:09.618663    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:18:11.212856 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:18:11.212780    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:18:12.820913 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:18:12.820876    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:18:14.504750 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:18:14.504715    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:18:15.996591 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:18:15.996557    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:18:17.713286 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:18:17.713244    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:18:19.440348 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:18:19.440166    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:18:19.974836 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:18:19.974796    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:18:21.220181 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:18:21.220132    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:18:22.919432 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:18:22.919395    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:18:24.618352 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:18:24.618317    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:18:26.316013 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:18:26.315969    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:18:28.015653 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:18:28.015617    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:18:28.712581 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:18:28.712532    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:18:29.763353 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:18:29.763317    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:18:31.258852 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:18:31.258814    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 23:18:31.420671 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:18:31.420639    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:18:33.113419 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:18:33.113382    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:18:33.884127 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:18:33.884090    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:18:34.736257 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:18:34.736199    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:18:36.408490 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:18:36.408457    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:18:38.089119 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:18:38.089060    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:18:39.444522 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:18:39.444490    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:18:39.616902 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:18:39.616863    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:18:40.624267 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:18:40.624209    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:18:40.765444 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:18:40.765407    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:18:41.264786 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:18:41.264756    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:18:47.337354 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:18:47.337319    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:18:47.484498 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:18:47.484460    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:18:47.691925 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:18:47.691884    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:18:52.381421 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:18:52.381387    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:18:52.942362 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:18:52.942327    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:18:53.379285 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:18:53.379189    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:18:57.471786 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:18:57.471746    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:19:01.258231 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:19:01.258181    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 23:19:10.459275 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:19:10.459209    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:19:11.584709 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:19:11.584670    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:19:11.864557 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:19:11.864478    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:19:24.940715 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:19:24.940678    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:19:25.083407 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:19:25.083371    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:19:26.435994 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:19:26.435955    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:19:31.258350 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:19:31.258312    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 23:19:35.105371 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:19:35.105333    4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-gvf4z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 23:19:35.106047 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:19:35.105997    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gvf4z" podUID="01158a33-90af-45e2-bf92-00a165ae2ac3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:19:41.960953 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:19:41.960874    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:19:42.929933 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:19:42.929896    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:19:50.967427 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:19:50.967375    4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-zfx74 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 23:19:50.968567 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:19:50.968118    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zfx74" podUID="1cee6d2b-00f8-425f-a4ec-75d80d3844d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:19:52.507556 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:19:52.507523    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:19:53.492503 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:19:53.492467    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:19:57.592868 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:19:57.592825    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:20:01.257894 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:20:01.257848    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 23:20:03.999081 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:20:03.999032    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:20:04.146817 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:20:04.146779    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:20:05.097746 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:20:05.097713    4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-gvf4z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 23:20:05.098449 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:20:05.098415    4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gvf4z" podUID="01158a33-90af-45e2-bf92-00a165ae2ac3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:20:28.540470 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:20:28.540426    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:20:29.059477 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:20:29.059430    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5d897ffcf4-xhjqz"]
Nov 03 23:20:29.063373 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:20:29.063347    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5d897ffcf4-xhjqz"
Nov 03 23:20:29.101608 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:20:29.101576    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5d897ffcf4-xhjqz"]
Nov 03 23:20:29.149668 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:20:29.149634    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:20:29.164680 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:20:29.164651    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/040637d8-b14b-43cc-b045-dbf63e4d73e2-maas-api-tls\") pod \"maas-api-5d897ffcf4-xhjqz\" (UID: \"040637d8-b14b-43cc-b045-dbf63e4d73e2\") " pod="odh-ai-gateway-infra/maas-api-5d897ffcf4-xhjqz"
Nov 03 23:20:29.164875 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:20:29.164710    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/040637d8-b14b-43cc-b045-dbf63e4d73e2-metrics-certs\") pod \"maas-api-5d897ffcf4-xhjqz\" (UID: \"040637d8-b14b-43cc-b045-dbf63e4d73e2\") " pod="odh-ai-gateway-infra/maas-api-5d897ffcf4-xhjqz"
Nov 03 23:20:29.164875 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:20:29.164784    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/040637d8-b14b-43cc-b045-dbf63e4d73e2-openshift-service-ca\") pod \"maas-api-5d897ffcf4-xhjqz\" (UID: \"040637d8-b14b-43cc-b045-dbf63e4d73e2\") " pod="odh-ai-gateway-infra/maas-api-5d897ffcf4-xhjqz"
Nov 03 23:20:29.164875 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:20:29.164839    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spjx7\" (UniqueName: \"kubernetes.io/projected/040637d8-b14b-43cc-b045-dbf63e4d73e2-kube-api-access-spjx7\") pod \"maas-api-5d897ffcf4-xhjqz\" (UID: \"040637d8-b14b-43cc-b045-dbf63e4d73e2\") " pod="odh-ai-gateway-infra/maas-api-5d897ffcf4-xhjqz"
Nov 03 23:20:29.265789 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:20:29.265756    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/040637d8-b14b-43cc-b045-dbf63e4d73e2-metrics-certs\") pod \"maas-api-5d897ffcf4-xhjqz\" (UID: \"040637d8-b14b-43cc-b045-dbf63e4d73e2\") " pod="odh-ai-gateway-infra/maas-api-5d897ffcf4-xhjqz"
Nov 03 23:20:29.265985 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:20:29.265806    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/040637d8-b14b-43cc-b045-dbf63e4d73e2-openshift-service-ca\") pod \"maas-api-5d897ffcf4-xhjqz\" (UID: \"040637d8-b14b-43cc-b045-dbf63e4d73e2\") " pod="odh-ai-gateway-infra/maas-api-5d897ffcf4-xhjqz"
Nov 03 23:20:29.265985 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:20:29.265854    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spjx7\" (UniqueName: \"kubernetes.io/projected/040637d8-b14b-43cc-b045-dbf63e4d73e2-kube-api-access-spjx7\") pod \"maas-api-5d897ffcf4-xhjqz\" (UID: \"040637d8-b14b-43cc-b045-dbf63e4d73e2\") " pod="odh-ai-gateway-infra/maas-api-5d897ffcf4-xhjqz"
Nov 03 23:20:29.265985 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:20:29.265887    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/040637d8-b14b-43cc-b045-dbf63e4d73e2-maas-api-tls\") pod \"maas-api-5d897ffcf4-xhjqz\" (UID: \"040637d8-b14b-43cc-b045-dbf63e4d73e2\") " pod="odh-ai-gateway-infra/maas-api-5d897ffcf4-xhjqz"
Nov 03 23:20:29.266627 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:20:29.266595    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/040637d8-b14b-43cc-b045-dbf63e4d73e2-openshift-service-ca\") pod \"maas-api-5d897ffcf4-xhjqz\" (UID: \"040637d8-b14b-43cc-b045-dbf63e4d73e2\") " pod="odh-ai-gateway-infra/maas-api-5d897ffcf4-xhjqz"
Nov 03 23:20:29.268422 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:20:29.268398    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/040637d8-b14b-43cc-b045-dbf63e4d73e2-maas-api-tls\") pod \"maas-api-5d897ffcf4-xhjqz\" (UID: \"040637d8-b14b-43cc-b045-dbf63e4d73e2\") " pod="odh-ai-gateway-infra/maas-api-5d897ffcf4-xhjqz"
Nov 03 23:20:29.268576 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:20:29.268553    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/040637d8-b14b-43cc-b045-dbf63e4d73e2-metrics-certs\") pod \"maas-api-5d897ffcf4-xhjqz\" (UID: \"040637d8-b14b-43cc-b045-dbf63e4d73e2\") " pod="odh-ai-gateway-infra/maas-api-5d897ffcf4-xhjqz"
Nov 03 23:20:29.288561 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:20:29.288534    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spjx7\" (UniqueName: \"kubernetes.io/projected/040637d8-b14b-43cc-b045-dbf63e4d73e2-kube-api-access-spjx7\") pod \"maas-api-5d897ffcf4-xhjqz\" (UID: \"040637d8-b14b-43cc-b045-dbf63e4d73e2\") " pod="odh-ai-gateway-infra/maas-api-5d897ffcf4-xhjqz"
Nov 03 23:20:29.315092 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:20:29.315015    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:20:29.376686 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:20:29.376647    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5d897ffcf4-xhjqz"
Nov 03 23:20:29.685586 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:20:29.685546    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5d897ffcf4-xhjqz"]
Nov 03 23:20:29.690871 ip-10-0-139-73 kubenswrapper[4431]: W1103 23:20:29.690842    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod040637d8_b14b_43cc_b045_dbf63e4d73e2.slice/crio-07212c7664ad5a11cf456c9caef6000d0c934cb8d64a0f1b40ca2590ce66c605 WatchSource:0}: Error finding container 07212c7664ad5a11cf456c9caef6000d0c934cb8d64a0f1b40ca2590ce66c605: Status 404 returned error can't find the container with id 07212c7664ad5a11cf456c9caef6000d0c934cb8d64a0f1b40ca2590ce66c605
Nov 03 23:20:29.692251 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:20:29.692204    4431 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 23:20:30.451464 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:20:30.451419    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5d897ffcf4-xhjqz" event={"ID":"040637d8-b14b-43cc-b045-dbf63e4d73e2","Type":"ContainerStarted","Data":"0f5071cfa4514b145eaaaa34ddad7c74fd17fd012e8e54460677bc88e1c15532"}
Nov 03 23:20:30.451702 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:20:30.451513    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5d897ffcf4-xhjqz" event={"ID":"040637d8-b14b-43cc-b045-dbf63e4d73e2","Type":"ContainerStarted","Data":"07212c7664ad5a11cf456c9caef6000d0c934cb8d64a0f1b40ca2590ce66c605"}
Nov 03 23:20:30.451702 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:20:30.451597    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5d897ffcf4-xhjqz"
Nov 03 23:20:30.494387 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:20:30.494335    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5d897ffcf4-xhjqz" podStartSLOduration=1.16972614 podStartE2EDuration="1.49432022s" totalImagesPullingTime="324.59408ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:20:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:20:30.493280994 +0000 UTC m=+4054.519136856" watchObservedRunningTime="2026-11-03 23:20:30.49432022 +0000 UTC m=+4054.520176081"
Nov 03 23:20:31.258897 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:20:31.258857    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 23:20:31.998365 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:20:31.998314    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:20:32.138550 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:20:32.138515    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:20:33.601018 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:20:33.600983    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:20:33.753320 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:20:33.753253    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:20:33.907667 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:20:33.907591    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:20:34.417924 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:20:34.417890    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:20:34.570825 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:20:34.570788    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:20:35.615349 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:20:35.615316    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:20:36.462380 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:20:36.462350    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5d897ffcf4-xhjqz"
Nov 03 23:20:36.539148 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:20:36.539114    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-84f7fbd778-pzqvs"]
Nov 03 23:20:36.539584 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:20:36.539554    4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-84f7fbd778-pzqvs" podUID="a974d9ab-19cd-448f-99ba-dbfe4446c562" containerName="maas-api" containerID="cri-o://0914caca3a0d3ac97f522e2c1ecbb2d7a3f1ef33ea1fa8b8e3672c905679fb9a" gracePeriod=30
Nov 03 23:20:37.210786 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:20:37.210753    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:20:37.796427 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:20:37.796403    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-84f7fbd778-pzqvs"
Nov 03 23:20:37.860332 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:20:37.860303    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a974d9ab-19cd-448f-99ba-dbfe4446c562-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a974d9ab-19cd-448f-99ba-dbfe4446c562-openshift-service-ca\") pod \"a974d9ab-19cd-448f-99ba-dbfe4446c562\" (UID: \"a974d9ab-19cd-448f-99ba-dbfe4446c562\") "
Nov 03 23:20:37.860485 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:20:37.860345    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a974d9ab-19cd-448f-99ba-dbfe4446c562-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a974d9ab-19cd-448f-99ba-dbfe4446c562-maas-api-tls\") pod \"a974d9ab-19cd-448f-99ba-dbfe4446c562\" (UID: \"a974d9ab-19cd-448f-99ba-dbfe4446c562\") "
Nov 03 23:20:37.860485 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:20:37.860469    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a974d9ab-19cd-448f-99ba-dbfe4446c562-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a974d9ab-19cd-448f-99ba-dbfe4446c562-metrics-certs\") pod \"a974d9ab-19cd-448f-99ba-dbfe4446c562\" (UID: \"a974d9ab-19cd-448f-99ba-dbfe4446c562\") "
Nov 03 23:20:37.860582 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:20:37.860510    4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a974d9ab-19cd-448f-99ba-dbfe4446c562-kube-api-access-2m4td\" (UniqueName: \"kubernetes.io/projected/a974d9ab-19cd-448f-99ba-dbfe4446c562-kube-api-access-2m4td\") pod \"a974d9ab-19cd-448f-99ba-dbfe4446c562\" (UID: \"a974d9ab-19cd-448f-99ba-dbfe4446c562\") "
Nov 03 23:20:37.860754 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:20:37.860728    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a974d9ab-19cd-448f-99ba-dbfe4446c562-openshift-service-ca" pod "a974d9ab-19cd-448f-99ba-dbfe4446c562" (UID: "a974d9ab-19cd-448f-99ba-dbfe4446c562"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:20:37.861020 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:20:37.861000    4431 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a974d9ab-19cd-448f-99ba-dbfe4446c562-openshift-service-ca\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:20:37.862651 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:20:37.862619    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a974d9ab-19cd-448f-99ba-dbfe4446c562-metrics-certs" pod "a974d9ab-19cd-448f-99ba-dbfe4446c562" (UID: "a974d9ab-19cd-448f-99ba-dbfe4446c562"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:20:37.863002 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:20:37.862973    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a974d9ab-19cd-448f-99ba-dbfe4446c562-maas-api-tls" pod "a974d9ab-19cd-448f-99ba-dbfe4446c562" (UID: "a974d9ab-19cd-448f-99ba-dbfe4446c562"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:20:37.863120 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:20:37.863088    4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a974d9ab-19cd-448f-99ba-dbfe4446c562-kube-api-access-2m4td" pod "a974d9ab-19cd-448f-99ba-dbfe4446c562" (UID: "a974d9ab-19cd-448f-99ba-dbfe4446c562"). InnerVolumeSpecName "kube-api-access-2m4td". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:20:37.962592 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:20:37.962556    4431 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a974d9ab-19cd-448f-99ba-dbfe4446c562-metrics-certs\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:20:37.962592 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:20:37.962586    4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2m4td\" (UniqueName: \"kubernetes.io/projected/a974d9ab-19cd-448f-99ba-dbfe4446c562-kube-api-access-2m4td\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:20:37.962592 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:20:37.962596    4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a974d9ab-19cd-448f-99ba-dbfe4446c562-maas-api-tls\") on node \"ip-10-0-139-73.ec2.internal\" DevicePath \"\""
Nov 03 23:20:38.487879 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:20:38.487844    4431 generic.go:356] "Generic (PLEG): container finished" podID="a974d9ab-19cd-448f-99ba-dbfe4446c562" containerID="0914caca3a0d3ac97f522e2c1ecbb2d7a3f1ef33ea1fa8b8e3672c905679fb9a" exitCode=0
Nov 03 23:20:38.488366 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:20:38.487899    4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-84f7fbd778-pzqvs"
Nov 03 23:20:38.488366 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:20:38.487938    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-84f7fbd778-pzqvs" event={"ID":"a974d9ab-19cd-448f-99ba-dbfe4446c562","Type":"ContainerDied","Data":"0914caca3a0d3ac97f522e2c1ecbb2d7a3f1ef33ea1fa8b8e3672c905679fb9a"}
Nov 03 23:20:38.488366 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:20:38.487995    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-84f7fbd778-pzqvs" event={"ID":"a974d9ab-19cd-448f-99ba-dbfe4446c562","Type":"ContainerDied","Data":"aabb256be130eee78e5cd97c24a7b08ec0b7ba3ab8f8a7a3e1744316dc8250b2"}
Nov 03 23:20:38.488366 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:20:38.488025    4431 scope.go:122] "RemoveContainer" containerID="0914caca3a0d3ac97f522e2c1ecbb2d7a3f1ef33ea1fa8b8e3672c905679fb9a"
Nov 03 23:20:38.496985 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:20:38.496971    4431 scope.go:122] "RemoveContainer" containerID="0914caca3a0d3ac97f522e2c1ecbb2d7a3f1ef33ea1fa8b8e3672c905679fb9a"
Nov 03 23:20:38.497276 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:20:38.497256    4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0914caca3a0d3ac97f522e2c1ecbb2d7a3f1ef33ea1fa8b8e3672c905679fb9a\": container with ID starting with 0914caca3a0d3ac97f522e2c1ecbb2d7a3f1ef33ea1fa8b8e3672c905679fb9a not found: ID does not exist" containerID="0914caca3a0d3ac97f522e2c1ecbb2d7a3f1ef33ea1fa8b8e3672c905679fb9a"
Nov 03 23:20:38.497322 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:20:38.497286    4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0914caca3a0d3ac97f522e2c1ecbb2d7a3f1ef33ea1fa8b8e3672c905679fb9a"} err="failed to get container status \"0914caca3a0d3ac97f522e2c1ecbb2d7a3f1ef33ea1fa8b8e3672c905679fb9a\": rpc error: code = NotFound desc = could not find container \"0914caca3a0d3ac97f522e2c1ecbb2d7a3f1ef33ea1fa8b8e3672c905679fb9a\": container with ID starting with 0914caca3a0d3ac97f522e2c1ecbb2d7a3f1ef33ea1fa8b8e3672c905679fb9a not found: ID does not exist"
Nov 03 23:20:38.537830 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:20:38.537802    4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-84f7fbd778-pzqvs"]
Nov 03 23:20:38.549526 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:20:38.549497    4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-84f7fbd778-pzqvs"]
Nov 03 23:20:38.936476 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:20:38.936438    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:20:40.086090 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:20:40.086047    4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a974d9ab-19cd-448f-99ba-dbfe4446c562" path="/var/lib/kubelet/pods/a974d9ab-19cd-448f-99ba-dbfe4446c562/volumes"
Nov 03 23:20:40.511080 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:20:40.511048    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:20:42.145842 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:20:42.145777    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:20:43.825799 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:20:43.825752    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:20:45.401356 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:20:45.401320    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:20:47.005721 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:20:47.005686    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:20:48.837209 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:20:48.837154    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:20:50.578800 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:20:50.578763    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:20:52.129091 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:20:52.129059    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:20:52.645593 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:20:52.645559    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:20:53.616305 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:20:53.616267    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:20:53.816795 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:20:53.816758    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:20:55.512577 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:20:55.512543    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:20:57.151330 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:20:57.151297    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:20:57.723765 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:20:57.723728    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:20:58.357936 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:20:58.357903    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:20:58.496094 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:20:58.496060    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:20:58.845803 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:20:58.845763    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:21:00.203377 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:21:00.203336    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:21:00.700371 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:21:00.700337    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:21:00.847883 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:21:00.847853    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:21:01.258191 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:21:01.258154    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 23:21:04.098484 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:21:04.098445    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:21:05.365287 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:21:05.365255    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:21:05.834628 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:21:05.834583    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:21:05.995208 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:21:05.995170    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:21:08.911333 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:21:08.911297    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:21:09.047747 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:21:09.047714    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:21:10.440765 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:21:10.440680    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:21:10.643007 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:21:10.642973    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:21:10.785937 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:21:10.785856    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:21:12.324373 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:21:12.324335    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:21:13.922932 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:21:13.922897    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:21:15.557561 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:21:15.557520    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:21:17.214144 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:21:17.214112    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:21:18.906961 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:21:18.906928    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:21:20.606040 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:21:20.605992    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:21:22.208139 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:21:22.208105    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:21:23.888420 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:21:23.888377    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:21:24.042622 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:21:24.042586    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:21:25.621958 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:21:25.621905    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:21:27.322786 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:21:27.322746    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:21:28.917037 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:21:28.916984    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:21:30.610658 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:21:30.610618    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:21:31.259484 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:21:31.259449    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 23:21:32.205704 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:21:32.205667    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:21:34.008875 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:21:34.008804    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:21:35.613628 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:21:35.613591    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:21:37.214775 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:21:37.214737    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:21:38.815119 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:21:38.815080    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:21:40.413916 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:21:40.413885    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:21:42.110432 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:21:42.110401    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:21:43.715700 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:21:43.715662    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:21:45.325423 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:21:45.325387    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:21:47.001686 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:21:47.001649    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:21:48.616095 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:21:48.616061    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:21:50.218127 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:21:50.218038    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:21:51.911242 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:21:51.911189    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:21:52.781637 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:21:52.781595    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:21:53.506714 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:21:53.506672    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:21:53.744172 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:21:53.744124    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:21:55.109516 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:21:55.109484    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:21:56.808727 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:21:56.808670    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:21:57.846178 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:21:57.846143    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:21:58.409925 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:21:58.409891    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:22:00.112839 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:22:00.112805    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:22:01.258544 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:22:01.258503    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 23:22:01.822112 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:22:01.822069    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:22:03.436896 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:22:03.436859    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:22:05.015035 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:22:05.014984    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:22:06.734503 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:22:06.734461    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:22:08.404849 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:22:08.404812    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:22:10.198840 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:22:10.198791    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:22:11.807413 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:22:11.807353    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:22:13.520735 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:22:13.520694    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:22:15.115482 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:22:15.115436    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:22:16.808230 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:22:16.808194    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:22:18.504321 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:22:18.504282    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:22:20.127010 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:22:20.126974    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:22:21.804189 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:22:21.804151    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:22:23.408928 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:22:23.408890    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:22:25.110069 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:22:25.110029    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:22:26.723930 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:22:26.723896    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:22:28.409550 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:22:28.409514    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:22:30.111776 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:22:30.111746    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:22:31.258520 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:22:31.258479    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 23:22:31.809119 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:22:31.809084    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:22:33.410367 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:22:33.410332    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:22:35.122274 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:22:35.122242    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:22:36.774021 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:22:36.773987    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:22:36.908732 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:22:36.908699    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:22:38.513660 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:22:38.513628    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:22:40.217134 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:22:40.217048    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:22:41.918298 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:22:41.918263    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:22:43.511712 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:22:43.511662    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:22:45.255741 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:22:45.255708    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:22:46.913713 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:22:46.913677    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:22:48.710092 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:22:48.710060    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:22:50.316112 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:22:50.316075    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:22:51.915746 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:22:51.915706    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:22:52.900184 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:22:52.900147    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:22:53.501563 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:22:53.501517    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:22:53.862441 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:22:53.862337    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:22:54.368113 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:22:54.368078    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:22:54.567815 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:22:54.567782    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:22:56.595910 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:22:56.595860    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:22:57.042049 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:22:57.042023    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mx5q8_ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea/ovn-acl-logging/0.log"
Nov 03 23:22:57.053648 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:22:57.053623    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mx5q8_ba86cda4-1b8a-413e-b7ae-0fa9b281d0ea/ovn-acl-logging/0.log"
Nov 03 23:22:57.967510 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:22:57.967481    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:23:01.258234 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:23:01.258179    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 23:23:01.505805 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:23:01.505774    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:23:01.996944 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:23:01.996908    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:23:02.135043 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:23:02.135009    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:23:02.763106 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:23:02.763061    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:23:08.936648 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:23:08.936602    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:23:09.079125 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:23:09.079084    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:23:09.292493 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:23:09.292397    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:23:17.078096 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:23:17.078059    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:23:31.258735 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:23:31.258693    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 23:23:35.459891 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:23:35.459856    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:23:46.996626 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:23:46.996589    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:23:53.021048 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:23:53.021010    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:23:53.981131 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:23:53.981081    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:23:58.086821 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:23:58.086780    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:24:01.257877 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:24:01.257836    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 23:24:16.741652 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:24:16.741576    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-h52bb_9b824440-4f91-46d4-beb5-a081da92aa4d/manager/0.log"
Nov 03 23:24:17.172632 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:24:17.172606    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-flhrd_21a175d3-b689-4352-91cb-dba3fe6f0247/manager/0.log"
Nov 03 23:24:17.292253 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:24:17.292191    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:24:17.676683 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:24:17.676650    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-rtdd4_f1b5a649-dc8d-48f5-9308-5a3370e5afcf/server/0.log"
Nov 03 23:24:17.868934 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:24:17.868903    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-sj75c_0d02a958-18da-44d7-a774-29e918540e52/manager/0.log"
Nov 03 23:24:18.099889 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:24:18.099809    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-kdpwc_202b2e09-563a-4ff1-bff1-edb9c7cc3d18/proxy/0.log"
Nov 03 23:24:19.040715 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:24:19.040681    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5d897ffcf4-xhjqz_040637d8-b14b-43cc-b045-dbf63e4d73e2/maas-api/0.log"
Nov 03 23:24:22.426156 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:24:22.426126    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-82jtt_8910869a-df66-49d2-a6b0-2a98eda07ea2/kuadrant-console-plugin/0.log"
Nov 03 23:24:24.474363 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:24:24.474331    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6b75ff7d64-48pqm_970868dd-c8fb-4960-a494-d76890bc0ff6/kube-auth-proxy/0.log"
Nov 03 23:24:25.234166 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:24:25.234136    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7df854bf67-fxflc_b051f82e-d473-415b-bba2-a659b94a106f/istio-proxy/0.log"
Nov 03 23:24:25.610877 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:24:25.610807    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7df854bf67-gvf4z_01158a33-90af-45e2-bf92-00a165ae2ac3/istio-proxy/0.log"
Nov 03 23:24:26.012493 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:24:26.012461    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7df854bf67-w85lm_f80ff4c3-aff2-4ea7-89f8-ae103660fdc9/istio-proxy/0.log"
Nov 03 23:24:26.415600 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:24:26.415563    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7df854bf67-zfx74_1cee6d2b-00f8-425f-a4ec-75d80d3844d9/istio-proxy/0.log"
Nov 03 23:24:31.257662 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:24:31.257621    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 23:24:39.466047 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:24:39.466016    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-h52bb_9b824440-4f91-46d4-beb5-a081da92aa4d/manager/0.log"
Nov 03 23:24:39.897300 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:24:39.897189    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-flhrd_21a175d3-b689-4352-91cb-dba3fe6f0247/manager/0.log"
Nov 03 23:24:40.363034 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:24:40.363003    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-rtdd4_f1b5a649-dc8d-48f5-9308-5a3370e5afcf/server/0.log"
Nov 03 23:24:40.563128 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:24:40.563092    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-sj75c_0d02a958-18da-44d7-a774-29e918540e52/manager/0.log"
Nov 03 23:24:40.791676 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:24:40.791627    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-kdpwc_202b2e09-563a-4ff1-bff1-edb9c7cc3d18/proxy/0.log"
Nov 03 23:24:41.541074 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:24:41.541044    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5d897ffcf4-xhjqz_040637d8-b14b-43cc-b045-dbf63e4d73e2/maas-api/0.log"
Nov 03 23:24:44.491991 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:24:44.491961    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-82jtt_8910869a-df66-49d2-a6b0-2a98eda07ea2/kuadrant-console-plugin/0.log"
Nov 03 23:24:46.088925 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:24:46.088895    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6b75ff7d64-48pqm_970868dd-c8fb-4960-a494-d76890bc0ff6/kube-auth-proxy/0.log"
Nov 03 23:24:46.848118 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:24:46.848074    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7df854bf67-fxflc_b051f82e-d473-415b-bba2-a659b94a106f/istio-proxy/0.log"
Nov 03 23:24:47.249950 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:24:47.249916    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7df854bf67-gvf4z_01158a33-90af-45e2-bf92-00a165ae2ac3/istio-proxy/0.log"
Nov 03 23:24:47.655573 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:24:47.655489    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7df854bf67-w85lm_f80ff4c3-aff2-4ea7-89f8-ae103660fdc9/istio-proxy/0.log"
Nov 03 23:24:48.063032 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:24:48.063002    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7df854bf67-zfx74_1cee6d2b-00f8-425f-a4ec-75d80d3844d9/istio-proxy/0.log"
Nov 03 23:24:53.149763 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:24:53.149716    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:24:54.104646 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:24:54.104608    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:24:58.208473 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:24:58.208426    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:25:01.259013 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:25:01.258978    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 23:25:31.257559 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:25:31.257520    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 23:25:53.305717 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:25:53.305619    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:25:54.273950 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:25:54.273897    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:25:58.327737 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:25:58.327693    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:26:01.258197 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:26:01.258160    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 23:26:11.519657 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:26:11.519610    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:26:11.667498 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:26:11.667465    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:26:15.651528 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:26:15.651494    4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log" limit=1024
Nov 03 23:26:16.043579 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:26:16.043548    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-66m8v_8ea0b360-1d72-4d40-935e-1394fecad74c/global-pull-secret-syncer/0.log"
Nov 03 23:26:17.013935 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:26:17.013905    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-7p8pl_8187068e-17ec-4e6d-bb8e-1d0312aa6674/konnectivity-agent/0.log"
Nov 03 23:26:17.880965 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:26:17.880936    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-73.ec2.internal_a7264ab43e0f926a44edf6ae184aafb2/haproxy/0.log"
Nov 03 23:26:24.775833 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:26:24.775802    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-82jtt_8910869a-df66-49d2-a6b0-2a98eda07ea2/kuadrant-console-plugin/0.log"
Nov 03 23:26:26.254893 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:26:26.254859    4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qtpzb/perf-node-gather-daemonset-p2n5k"]
Nov 03 23:26:26.255426 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:26:26.255317    4431 cpu_manager.go:414] "containerMap: removing container" podUID="a974d9ab-19cd-448f-99ba-dbfe4446c562" containerName="maas-api"
Nov 03 23:26:26.255426 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:26:26.255336    4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a974d9ab-19cd-448f-99ba-dbfe4446c562" containerName="maas-api"
Nov 03 23:26:26.255507 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:26:26.255454    4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="a974d9ab-19cd-448f-99ba-dbfe4446c562" containerName="maas-api"
Nov 03 23:26:26.258686 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:26:26.258670    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qtpzb/perf-node-gather-daemonset-p2n5k"
Nov 03 23:26:26.277288 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:26:26.277258    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qtpzb/perf-node-gather-daemonset-p2n5k"]
Nov 03 23:26:26.347020 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:26:26.346983    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a47cb2ec-340b-4e23-8bef-6dc1abacc4d1-lib-modules\") pod \"perf-node-gather-daemonset-p2n5k\" (UID: \"a47cb2ec-340b-4e23-8bef-6dc1abacc4d1\") " pod="openshift-must-gather-qtpzb/perf-node-gather-daemonset-p2n5k"
Nov 03 23:26:26.347020 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:26:26.347026    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a47cb2ec-340b-4e23-8bef-6dc1abacc4d1-proc\") pod \"perf-node-gather-daemonset-p2n5k\" (UID: \"a47cb2ec-340b-4e23-8bef-6dc1abacc4d1\") " pod="openshift-must-gather-qtpzb/perf-node-gather-daemonset-p2n5k"
Nov 03 23:26:26.347288 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:26:26.347054    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a47cb2ec-340b-4e23-8bef-6dc1abacc4d1-podres\") pod \"perf-node-gather-daemonset-p2n5k\" (UID: \"a47cb2ec-340b-4e23-8bef-6dc1abacc4d1\") " pod="openshift-must-gather-qtpzb/perf-node-gather-daemonset-p2n5k"
Nov 03 23:26:26.347288 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:26:26.347203    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a47cb2ec-340b-4e23-8bef-6dc1abacc4d1-sys\") pod \"perf-node-gather-daemonset-p2n5k\" (UID: \"a47cb2ec-340b-4e23-8bef-6dc1abacc4d1\") " pod="openshift-must-gather-qtpzb/perf-node-gather-daemonset-p2n5k"
Nov 03 23:26:26.347374 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:26:26.347292    4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psb2v\" (UniqueName: \"kubernetes.io/projected/a47cb2ec-340b-4e23-8bef-6dc1abacc4d1-kube-api-access-psb2v\") pod \"perf-node-gather-daemonset-p2n5k\" (UID: \"a47cb2ec-340b-4e23-8bef-6dc1abacc4d1\") " pod="openshift-must-gather-qtpzb/perf-node-gather-daemonset-p2n5k"
Nov 03 23:26:26.366464 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:26:26.366443    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qtpzb\"/\"openshift-service-ca.crt\""
Nov 03 23:26:26.366666 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:26:26.366649    4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qtpzb\"/\"kube-root-ca.crt\""
Nov 03 23:26:26.367068 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:26:26.367054    4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-qtpzb\"/\"default-dockercfg-8d4gm\""
Nov 03 23:26:26.448013 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:26:26.447979    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a47cb2ec-340b-4e23-8bef-6dc1abacc4d1-podres\") pod \"perf-node-gather-daemonset-p2n5k\" (UID: \"a47cb2ec-340b-4e23-8bef-6dc1abacc4d1\") " pod="openshift-must-gather-qtpzb/perf-node-gather-daemonset-p2n5k"
Nov 03 23:26:26.448184 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:26:26.448033    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a47cb2ec-340b-4e23-8bef-6dc1abacc4d1-sys\") pod \"perf-node-gather-daemonset-p2n5k\" (UID: \"a47cb2ec-340b-4e23-8bef-6dc1abacc4d1\") " pod="openshift-must-gather-qtpzb/perf-node-gather-daemonset-p2n5k"
Nov 03 23:26:26.448184 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:26:26.448073    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psb2v\" (UniqueName: \"kubernetes.io/projected/a47cb2ec-340b-4e23-8bef-6dc1abacc4d1-kube-api-access-psb2v\") pod \"perf-node-gather-daemonset-p2n5k\" (UID: \"a47cb2ec-340b-4e23-8bef-6dc1abacc4d1\") " pod="openshift-must-gather-qtpzb/perf-node-gather-daemonset-p2n5k"
Nov 03 23:26:26.448184 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:26:26.448122    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a47cb2ec-340b-4e23-8bef-6dc1abacc4d1-podres\") pod \"perf-node-gather-daemonset-p2n5k\" (UID: \"a47cb2ec-340b-4e23-8bef-6dc1abacc4d1\") " pod="openshift-must-gather-qtpzb/perf-node-gather-daemonset-p2n5k"
Nov 03 23:26:26.448184 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:26:26.448153    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a47cb2ec-340b-4e23-8bef-6dc1abacc4d1-lib-modules\") pod \"perf-node-gather-daemonset-p2n5k\" (UID: \"a47cb2ec-340b-4e23-8bef-6dc1abacc4d1\") " pod="openshift-must-gather-qtpzb/perf-node-gather-daemonset-p2n5k"
Nov 03 23:26:26.448418 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:26:26.448184    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a47cb2ec-340b-4e23-8bef-6dc1abacc4d1-sys\") pod \"perf-node-gather-daemonset-p2n5k\" (UID: \"a47cb2ec-340b-4e23-8bef-6dc1abacc4d1\") " pod="openshift-must-gather-qtpzb/perf-node-gather-daemonset-p2n5k"
Nov 03 23:26:26.448418 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:26:26.448189    4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a47cb2ec-340b-4e23-8bef-6dc1abacc4d1-proc\") pod \"perf-node-gather-daemonset-p2n5k\" (UID: \"a47cb2ec-340b-4e23-8bef-6dc1abacc4d1\") " pod="openshift-must-gather-qtpzb/perf-node-gather-daemonset-p2n5k"
Nov 03 23:26:26.448418 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:26:26.448264    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a47cb2ec-340b-4e23-8bef-6dc1abacc4d1-proc\") pod \"perf-node-gather-daemonset-p2n5k\" (UID: \"a47cb2ec-340b-4e23-8bef-6dc1abacc4d1\") " pod="openshift-must-gather-qtpzb/perf-node-gather-daemonset-p2n5k"
Nov 03 23:26:26.448418 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:26:26.448311    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a47cb2ec-340b-4e23-8bef-6dc1abacc4d1-lib-modules\") pod \"perf-node-gather-daemonset-p2n5k\" (UID: \"a47cb2ec-340b-4e23-8bef-6dc1abacc4d1\") " pod="openshift-must-gather-qtpzb/perf-node-gather-daemonset-p2n5k"
Nov 03 23:26:26.470958 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:26:26.470927    4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psb2v\" (UniqueName: \"kubernetes.io/projected/a47cb2ec-340b-4e23-8bef-6dc1abacc4d1-kube-api-access-psb2v\") pod \"perf-node-gather-daemonset-p2n5k\" (UID: \"a47cb2ec-340b-4e23-8bef-6dc1abacc4d1\") " pod="openshift-must-gather-qtpzb/perf-node-gather-daemonset-p2n5k"
Nov 03 23:26:26.571849 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:26:26.571774    4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qtpzb/perf-node-gather-daemonset-p2n5k"
Nov 03 23:26:26.643768 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:26:26.643744    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_797dcec6-675d-4af2-89a4-1b5902ae547a/alertmanager/0.log"
Nov 03 23:26:26.770326 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:26:26.770299    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_797dcec6-675d-4af2-89a4-1b5902ae547a/config-reloader/0.log"
Nov 03 23:26:26.870606 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:26:26.870528    4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qtpzb/perf-node-gather-daemonset-p2n5k"]
Nov 03 23:26:26.875421 ip-10-0-139-73 kubenswrapper[4431]: W1103 23:26:26.875396    4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda47cb2ec_340b_4e23_8bef_6dc1abacc4d1.slice/crio-75f4548530a9e6fa7454ab5f6e9c118b2c7225aa7b02e36c7205f684d53b3a36 WatchSource:0}: Error finding container 75f4548530a9e6fa7454ab5f6e9c118b2c7225aa7b02e36c7205f684d53b3a36: Status 404 returned error can't find the container with id 75f4548530a9e6fa7454ab5f6e9c118b2c7225aa7b02e36c7205f684d53b3a36
Nov 03 23:26:26.890949 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:26:26.890929    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_797dcec6-675d-4af2-89a4-1b5902ae547a/kube-rbac-proxy-web/0.log"
Nov 03 23:26:27.011990 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:26:27.011967    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_797dcec6-675d-4af2-89a4-1b5902ae547a/kube-rbac-proxy/0.log"
Nov 03 23:26:27.133411 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:26:27.133327    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_797dcec6-675d-4af2-89a4-1b5902ae547a/kube-rbac-proxy-metric/0.log"
Nov 03 23:26:27.136534 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:26:27.136509    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qtpzb/perf-node-gather-daemonset-p2n5k" event={"ID":"a47cb2ec-340b-4e23-8bef-6dc1abacc4d1","Type":"ContainerStarted","Data":"2ac0c038c96676dc349dae16c9bc5f4950baef1e88232c3b29ae147fd122fef1"}
Nov 03 23:26:27.136680 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:26:27.136559    4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qtpzb/perf-node-gather-daemonset-p2n5k" event={"ID":"a47cb2ec-340b-4e23-8bef-6dc1abacc4d1","Type":"ContainerStarted","Data":"75f4548530a9e6fa7454ab5f6e9c118b2c7225aa7b02e36c7205f684d53b3a36"}
Nov 03 23:26:27.136680 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:26:27.136657    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-qtpzb/perf-node-gather-daemonset-p2n5k"
Nov 03 23:26:27.177684 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:26:27.177634    4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-qtpzb/perf-node-gather-daemonset-p2n5k" podStartSLOduration=1.177610812 podStartE2EDuration="1.177610812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 23:26:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:26:27.175033311 +0000 UTC m=+4411.200889174" watchObservedRunningTime="2026-11-03 23:26:27.177610812 +0000 UTC m=+4411.203466677"
Nov 03 23:26:27.258650 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:26:27.258622    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_797dcec6-675d-4af2-89a4-1b5902ae547a/prom-label-proxy/0.log"
Nov 03 23:26:27.379158 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:26:27.379132    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_797dcec6-675d-4af2-89a4-1b5902ae547a/init-config-reloader/0.log"
Nov 03 23:26:29.019020 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:26:29.018989    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mq2bl_97666ecb-018e-46f6-9a80-c0beb294c7d6/node-exporter/0.log"
Nov 03 23:26:29.138629 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:26:29.138603    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mq2bl_97666ecb-018e-46f6-9a80-c0beb294c7d6/kube-rbac-proxy/0.log"
Nov 03 23:26:29.258115 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:26:29.258088    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mq2bl_97666ecb-018e-46f6-9a80-c0beb294c7d6/init-textfile/0.log"
Nov 03 23:26:29.781941 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:26:29.781910    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6e0203c3-558e-4e70-93ec-cd84edfe35e1/prometheus/0.log"
Nov 03 23:26:29.901589 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:26:29.901558    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6e0203c3-558e-4e70-93ec-cd84edfe35e1/config-reloader/0.log"
Nov 03 23:26:30.023015 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:26:30.022985    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6e0203c3-558e-4e70-93ec-cd84edfe35e1/thanos-sidecar/0.log"
Nov 03 23:26:30.142139 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:26:30.142064    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6e0203c3-558e-4e70-93ec-cd84edfe35e1/kube-rbac-proxy-web/0.log"
Nov 03 23:26:30.262101 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:26:30.262070    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6e0203c3-558e-4e70-93ec-cd84edfe35e1/kube-rbac-proxy/0.log"
Nov 03 23:26:30.381411 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:26:30.381376    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6e0203c3-558e-4e70-93ec-cd84edfe35e1/kube-rbac-proxy-thanos/0.log"
Nov 03 23:26:30.502331 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:26:30.502306    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6e0203c3-558e-4e70-93ec-cd84edfe35e1/init-config-reloader/0.log"
Nov 03 23:26:30.629148 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:26:30.629116    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-f7ksv_dfa4f1ab-8da9-412a-9e83-b655e65c4212/prometheus-operator/0.log"
Nov 03 23:26:30.743069 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:26:30.743037    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-f7ksv_dfa4f1ab-8da9-412a-9e83-b655e65c4212/kube-rbac-proxy/0.log"
Nov 03 23:26:30.864127 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:26:30.864051    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-lkstc_cdc51322-d4e3-4345-84f5-e52653531823/prometheus-operator-admission-webhook/0.log"
Nov 03 23:26:30.988560 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:26:30.988531    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-979f6bd4f-xgz4d_66a72772-83f1-4117-a856-f52c0ebbe966/telemeter-client/0.log"
Nov 03 23:26:31.113593 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:26:31.113563    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-979f6bd4f-xgz4d_66a72772-83f1-4117-a856-f52c0ebbe966/reload/0.log"
Nov 03 23:26:31.235395 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:26:31.235368    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-979f6bd4f-xgz4d_66a72772-83f1-4117-a856-f52c0ebbe966/kube-rbac-proxy/0.log"
Nov 03 23:26:31.258929 ip-10-0-139-73 kubenswrapper[4431]: E1103 23:26:31.258884    4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6ndx" containerName="extractor"
Nov 03 23:26:33.151736 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:26:33.151707    4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-qtpzb/perf-node-gather-daemonset-p2n5k"
Nov 03 23:26:35.836926 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:26:35.836896    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-gzwb4_623e390b-6a39-48cd-9d7c-4a5890c11c6e/download-server/0.log"
Nov 03 23:26:39.528389 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:26:39.528357    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4qrls_309c2503-3fef-4048-af01-6b3b70aa52a1/dns/0.log"
Nov 03 23:26:39.649349 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:26:39.649318    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4qrls_309c2503-3fef-4048-af01-6b3b70aa52a1/kube-rbac-proxy/0.log"
Nov 03 23:26:40.299078 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:26:40.299048    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-x7cpm_94868020-2e98-4295-8db5-043461cc6e9d/dns-node-resolver/0.log"
Nov 03 23:26:41.391204 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:26:41.391170    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-fc6bcdddf-9zl84_1bdb4a48-dd3c-4037-8130-78c55a7121b5/registry/0.log"
Nov 03 23:26:41.776695 ip-10-0-139-73 kubenswrapper[4431]: I1103 23:26:41.776669    4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-p4p25_cc66a3de-7e7e-4855-8f17-db97dcfa815a/node-ca/0.log"